How should a high-throughput service absorb provider 429 responses?

Attempt it

Your service regularly hits an LLM provider's rate limit and receives 429s. Describe how to absorb them well, and why blind immediate retries make it worse.

Free · 2 AI evals / day

Quick reference

TL;DR

Shape demand to the quota: honor the Retry-After header, smooth your send rate with a client-side token bucket, back off with jitter, and shed low-priority traffic near the cap — never retry a 429 immediately.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a popular ride at a theme park with a 'one group every 30 seconds' rule. If you shove your whole crowd at the gate the instant you're turned away, you just jam it harder and everyone's turned away again. The smart move is to let people through in a steady trickle that matches the gate's pace, and when the operator says 'wait 30 seconds', you actually wait. If the line is too long, you send the VIPs first and ask the casual visitors to come back later. That steady, signal-following pacing is exactly how a service should handle a provider that says 'too many requests'.

Key concepts

LLM System Design Inference Optimization Observability and Tracing

Quick scan · 2–4 min

Everything important, quickly.

Spend 6-8 minutes on proactive shaping versus reactive retries and the distributed-limiter trap, the two ideas that separate a senior answer from a junior one here.

Code example

python

# Honor Retry-After, fall back to jittered backoff
import random, time

def on_429(resp, attempt, base=0.5):
    ra = resp.headers.get("Retry-After")
    if ra is not None:
        wait = float(ra)                  # provider told us exactly
    else:
        wait = base * 2 ** attempt        # exponential backoff
        wait *= random.random()           # full jitter, de-sync clients
    time.sleep(wait)
    # ...then retry, but only if a token-bucket admit() also allows it,
    # so we stay under the quota instead of bursting back into it.

Real-world examples

Real products, models, and research that use this idea.

OpenAI and Anthropic APIs return a Retry-After header on 429s plus rate-limit headers showing remaining quota.
LLM gateways like LiteLLM and Portkey enforce client-side rate limits and queue requests to stay under provider caps.
Distributed token-bucket limiters backed by Redis are the standard way multi-instance services share a provider quota.
AWS SDKs use full-jitter exponential backoff as the reference pattern for absorbing throttling responses.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you size and share a token bucket across many service instances behind a load balancer?

Sign in or go Pro to see the approach for this follow-up.

QThe provider rate-limits on tokens per minute, not requests per minute. How does that change your limiter?

Sign in or go Pro to see the approach for this follow-up.

QHow do you keep retry storms from forming when an entire region of clients gets 429d at once?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

AWS Architecture Blog: Exponential Backoff And Jitter