How should a high throughput service absorb provider 429 responses?
Your service regularly hits an LLM provider's rate limit and receives 429s. Describe how to absorb them well, and why blind immediate retries make it worse.
Shape demand to the quota: honor the Retry-After header, smooth your send rate with a client-side token bucket, back off with jitter, and shed low-priority traffic near the cap — never retry a 429 immediately.
Imagine a popular ride at a theme park with a 'one group every 30 seconds' rule. If you shove your whole crowd at the gate the instant you're turned away, you just jam it harder and everyone's turned away again. The smart move is to let people through in a steady trickle that matches the gate's pace, and when the operator says 'wait 30 seconds', you actually wait. If the line is too long, you send the VIPs first and ask the casual visitors to come back later. That steady, signal-following pacing is exactly how a service should handle a provider that says 'too many requests'.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A 429 Too Many Requests is the provider telling you, politely, that you're sending faster than your quota allows. The instinctive engineering reaction — catch the error and retry — is exactly the wrong one, because a retry is just another request hitting a limit you've already exceeded. Done blindly and at scale, that reaction doesn't recover from the throttle; it amplifies it.
The right framing is that rate-limit handling is a demand-shaping problem, not an error-handling one. You have a fixed budget of requests (or tokens) per unit time, and your job is to fit your traffic inside it. That reframing changes where you put your effort: most of it goes upstream of the provider, smoothing and prioritizing traffic before it ever leaves your service, and only a little goes into reacting to the 429s that slip through.
This deep dive covers why immediate retries amplify rather than absorb, how a token bucket shapes outgoing rate, the distributed-limiter trap that bites multi-instance services, how to honor the provider's own Retry-After signal, and how to shed load by priority when you're genuinely up against the cap.
Why a blind retry amplifies instead of absorbing
Picture the moment you get a 429. By definition, you've just sent more requests in the window than the limit allows. The provider is already saying 'stop'.
Now you immediately retry. That retry is not a recovery action — it's literally one more request against the same exceeded limit. It can't succeed, because the condition that caused the 429 hasn't changed. All it does is add load to a provider that's already pushing back, and burn your own resources doing it.
The damage multiplies when many clients are involved. In a high-throughput service, a 429 rarely hits one request in isolation — a burst trips the limit and dozens or thousands of requests get throttled in the same instant. If they all retry immediately, they all hit the provider again at the same instant. The spike that caused the original throttle gets re-created, the limit stays pinned, and a problem that should have lasted a few hundred milliseconds stretches into minutes. This is a retry storm: synchronized clients hammering in lockstep, each retry keeping the breach alive for all the others.
So the first principle is counterintuitive but firm: when you're over a rate limit, the immediate priority is to send less, not to send the same thing again faster. Everything else in good 429 handling follows from that.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Honor Retry-After, fall back to jittered backoff
import random, time
def on_429(resp, attempt, base=0.5):
ra = resp.headers.get("Retry-After")
if ra is not None:
wait = float(ra) # provider told us exactly
else:
wait = base * 2 ** attempt # exponential backoff
wait *= random.random() # full jitter, de-sync clients
time.sleep(wait)
# ...then retry, but only if a token-bucket admit() also allows it,
# so we stay under the quota instead of bursting back into it.Real products, models, and research that use this idea.
- OpenAI and Anthropic APIs return a Retry-After header on 429s plus rate-limit headers showing remaining quota.
- LLM gateways like LiteLLM and Portkey enforce client-side rate limits and queue requests to stay under provider caps.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you size and share a token bucket across many service instances behind a load balancer?
Use a distributed limiter — a shared Redis counter or a sidecar — so the fleet draws from one quota; per-instance buckets each assume the full rate and collectively overshoot.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Retrying a 429 immediately in a tight loop, which is just another request against an already-exceeded limit and synchronizes every client into a retry storm.
60 second bullets to scan on the way to the call.
Why an immediate retry of a 429 deepens the breach
What the Retry-After header gives you over a guessed delay
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.