The bucket has tokens. The service is still overloaded.

One limit cannot answer three questions

A document-export API admits two requests each second. That sounds safe until each export occupies a database connection for twenty seconds. At a steady arrival rate, forty exports can be running together. The arrival limit is working while the database pool is disappearing.

Rate limiting has at least three separate jobs. A burst limit asks how much work may arrive at once. A sustained-rate limit asks how quickly that allowance returns. A concurrency limit asks how much admitted work may remain unfinished. Arpit Bhayani’s discussion of Stripe’s historical design makes this separation concrete: tenant rate limits, in-flight limits and load shedding protect different boundaries. Hello Interview’s walkthrough supplies the distributed token-bucket mechanics. The numbers below are our example, not measurements from either company.

A burst is capacity; a rate is refill

Give tenant acme a bucket for POST /exports with capacity six and refill rate two tokens per second. Six requests arriving together can start. A seventh request at the same instant is rejected. After 1.5 seconds, three tokens have returned, capped at six. This policy permits a short burst without granting six requests every second forever.

A fixed window such as “120 requests per minute” can admit 120 requests just before a boundary and another 120 just after it. A token bucket expresses the burst explicitly. It still needs a separate concurrency gate when request cost depends on duration.

The decision and the state change are one operation

Every gateway must make the same logical transition: refill from elapsed time, test the available balance, subtract the request’s cost, and save the new balance and timestamp. Splitting those steps into separate reads and writes lets two gateways spend the same last token.

atomic take(key, now, cost):
  t = max(now, last_refill)
  available = min(capacity, tokens + (t - last_refill) * rate)
  allowed = available >= cost
  tokens = available - (cost if allowed else 0)
  last_refill = t
  save(tokens, last_refill)
  return allowed

Persist the balance and timestamp together on both allowance and denial. Validate the request cost and bucket settings; a cost above capacity will never fit. A Redis script, a transactional database update or a purpose-built limiter can make that transition atomic for one key. The key defines the fairness boundary. tenant:acme:route:export isolates one tenant and route; an IP-only key groups unrelated users behind a NAT, while an account-only key may let one expensive route starve every other route. Global, tenant and endpoint budgets can be layered, but the request must pass all of them.

Returned reset metadata is also state. Arpit’s account of a GitHub rate-limit bug shows why combining a rounded TTL from the server with a separately sampled application clock can make a reset time move between responses. Store or derive reset information under one time model instead of assembling it after a network round trip.

Count work that has not finished

After the token decision, the export endpoint acquires one of eight concurrency permits. If none is available, it rejects or queues according to an explicit policy. Queuing is not free capacity: a bounded queue trades rejection for waiting and needs an age limit so stale exports do not consume the recovery period.

The permit must be released on success, error, timeout and cancellation. A process crash makes a purely in-memory distributed permit harder; a lease needs an expiry longer than normal work, renewal while work is healthy, and reconciliation for abandoned jobs. An expiring lease alone does not prove a hard concurrency cap: a partitioned worker may still be running after its lease expires. A strict cap needs enforcement where work executes, or confirmed cancellation before the permit is reissued. A worker-local admission limit can also protect each node.

Request cost can be weighted. A metadata lookup might spend one token and no scarce permit, while an export spends several tokens and one database permit. HTTP method alone does not tell you which operation matters or costs more.

Choose outage behavior before the store fails

If the shared limiter is unavailable, “fail open” preserves reachability but can remove the last protection before overload. “Fail closed” protects capacity but turns a limiter fault into an API outage. The choice belongs to each boundary: a public read may use a conservative local fallback, a password-attempt limit may deny when state is unknown, and an internal bulk export may pause while interactive work continues.

Keep the last valid rules with a version and expiry. Apply a small worker-local ceiling while shared state recovers. Emit separate metrics for policy denials, concurrency denials, fallback decisions and limiter errors; otherwise a successful rejection and a broken dependency look identical. Return an appropriate status such as 429 with usable retry guidance, and expect clients to add bounded jitter. A limiter is an overload policy, so its own failure mode must preserve the work the service values most.