Rate Limiting Is Admission Control: Rate, Concurrency, and Quotas
“Allow 100 requests per minute and return 429 after that” describes one counter. It does not say who shares those requests, whether they may arrive in one burst, how long-running work consumes capacity, how several instances agree, or what happens when the limiter’s storage fails. Production rate limiting is admission control first and an HTTP response second.
Do not collapse four budgets into one
- Rate bounds work per unit of time, such as 20 units per second.
- Burst is capacity that may accumulate and be spent at once.
- Concurrency bounds in-flight work to protect threads, pools, and downstreams.
- Quota is total consumption per day or billing period, often tied to cost or a plan.
A request can pass the rate check and occupy a concurrency slot for five minutes. It can have available concurrency after its monthly quota is exhausted. Model and observe these budgets separately, and identify the actual policy in an error. One “QPS” number cannot provide fairness and protect slow resources at the same time.
Limit by the business identity
For authenticated APIs, prefer a tenant, account, API key, or OAuth client, then refine by endpoint and resource. IP address is mainly a fallback for anonymous abuse. A company NAT can put thousands of users behind one address, while mobile users change addresses frequently. Use a client address only after a trusted proxy has replaced and sanitized forwarding headers; never trust an external X-Forwarded-For directly.
Admission is usually hierarchical: a global policy protects the service, a tenant policy preserves fairness, a principal policy prevents one user from dominating, and an endpoint policy bounds costly operations. A request must pass every applicable layer. Keys should contain internal or irreversible identifiers, not tokens or email addresses. High-cardinality anonymous keys also need expiry and a capacity bound.
The algorithm defines the allowed traffic shape
A fixed window is simple, but a client can consume one allowance at the end of a window and another at the start of the next. A sliding log is precise but stores each event; a sliding counter trades precision for space. A token bucket is often a useful API model: bucket capacity defines burst and refill rate defines sustained throughput.
available = min(capacity, available + elapsed * refill_rate)
if available >= cost:
available -= cost
allow
else:
deny
Measure elapsed time with a monotonic clock. Capacity refill and debit must happen in one atomic decision. The Redis counter example likewise notes that separating INCR from a first EXPIRE creates a race; a transaction or script can combine state changes. Reading a balance, performing business work, and charging afterward lets concurrent requests pass together.
A shared store can coordinate instances strongly, at the cost of one remote dependency per decision. If instances receive batches of local tokens to reduce latency, calculate the maximum overshoot—such as node count times allocation size—and ensure it remains acceptable during scaling. A given key must route to a stable decision point so switching regions cannot grant a full allowance twice.
One request does not always cost one unit
A health check and a large report do not have equal cost. Policies can debit units by endpoint, query complexity, input size, or predicted downstream calls. Calculation needs a ceiling and must happen before expensive work, or the limiter itself becomes an attack surface.
When true cost is known only afterward, reserve an upper bound and reconcile under an explicit rule. Refund behavior is both a product and abuse decision. A server failure might deserve a refund; invalid credentials, malicious parsing, or client cancellation might not. The rule must be predictable so retries cannot create unlimited free work.
Concurrency uses a bounded semaphore or lease that releases on completion, cancellation, and timeout. Streaming responses, WebSockets, and background exports need separate pools so a few long connections do not displace ordinary APIs. A lease needs ownership and expiry so a crashed instance cannot hold a slot forever.
429 is feedback, not the internal design
RFC 6585 defines 429 Too Many Requests but deliberately leaves user identification and counting to the server. Return a stable machine code, policy scope, and earliest useful retry time. Retry-After can be delay seconds or an HTTP date. Use 429 when a particular caller exceeded policy; use 503 Service Unavailable for service-wide overload or maintenance rather than labeling a capacity incident as client abuse.
HTTP/1.1 429 Too Many Requests
Content-Type: application/problem+json
Retry-After: 12
Cache-Control: no-store
As of this article’s date, the IETF HTTPAPI RateLimit fields remain an Internet-Draft, not a published RFC. If adopting its RateLimit-Policy and RateLimit structures, pin the implemented draft and document compatibility. Existing X-RateLimit-* conventions do not have consistent semantics, so third-party clients cannot safely infer them.
A client treats Retry-After as the earliest retry time and adds jitter. Batch work can slow or queue; an interactive request should fail quickly. A rejected non-idempotent operation must not have created a side effect. If that boundary cannot be guaranteed, use an idempotency key rather than relying on a client to retry only once.
Give limiter failure an explicit policy
Failing open whenever shared storage times out can overwhelm a downstream exactly when protection is needed. Failing closed everywhere turns an auxiliary component failure into a full outage. Decide by endpoint. Login and promotion issuance can fail closed; public reads can use a conservative local emergency bucket; critical internal writes might fail open briefly but retain a global concurrency fuse.
The failure path needs a short timeout, circuit breaker, and recovery probes so requests do not queue behind limiter storage. Any “remaining” value during degraded operation is approximate. Avoid exposing exact system-wide capacity that would help an attacker map the defense.
Run in shadow mode before rejecting
Record “would deny” first and compare impact across tenants, endpoints, and client versions. Track allow and deny, policy ID, principal class, consumed units, retry time, occupied slots, storage latency, degraded decisions, and overshoot bound without logging raw credentials. Watch whether blocked traffic simply moves to another endpoint.
Use virtual time to test window boundaries, burst depletion and refill, negative cost, and overflow. Concurrency tests should prove atomic debit never exceeds the bound. Integration tests need multiple nodes, storage partition, clock jumps, cancellation release, lease expiry, live policy updates, IPv6 and proxy addresses, plus clients that both honor and ignore Retry-After.
Good rate limiting is not about rejecting as many requests as possible. It establishes a defensible admission boundary across resources, cost, and fairness. Once identity, budgets, algorithms, failure modes, and client feedback are explicit, 429 becomes a useful protocol result instead of decoration after the system is already overloaded.