Every API request passes through a limiter at the gateway, which atomically updates a counter (or token bucket) for that client in a shared in-memory store and returns allow or deny. Rules live in a config database and are pushed out to every limiter instance, so changes apply in seconds without redeploys. If the counter store is down, the limiter fails open rather than taking the API down with it.
Functional requirements
- Limit requests per client identity (user, API key or IP), e.g. 100 per minute
- Different limits per endpoint, plan or tier
- Return HTTP 429 with Retry-After and X-RateLimit-* headers when over the limit
- Admins can add or change rules without redeploying
- Report usage and denials for monitoring and billing
Non-functional requirements
- Adds less than ~1–2 ms to each request
- Limits hold across many gateway instances (distributed, not per-server)
- Highly available; if the limiter fails, traffic keeps flowing (fail open)
- Small over-counting or under-counting at window edges is acceptable
Extensions
- Concurrency limits (in-flight requests) in addition to rate limits.
- Adaptive limits that tighten automatically when backends are overloaded.
- Quotas billed per plan, with usage exported to billing.
- Per-IP limits at the CDN/WAF layer to absorb floods before they reach the gateway.