Sequence a robust retry policy for a transient LLM API failure
- 1Fall back to a degraded path (cheaper model, cached, or graceful error)
- 2Wait an exponentially increasing delay with random jitter added
- 3Stop after a maximum attempt cap is reached
- 4Retry the call, incrementing the attempt counter
- 5Catch the failure and check whether the error is retryable (429 / 5xx / timeout)
- 6Set a per request timeout so a hung call can't block indefinitely
A robust retry wraps each call in a timeout, classifies the error as retryable, waits an exponential backoff plus jitter, retries up to a cap, then degrades — never a blind immediate retry loop.
Imagine knocking on a friend's door and nobody answers. Banging again and again the instant nobody opens just annoys everyone and never works faster. A smart visitor waits a bit, then a little longer, and adds a random pause so a whole crowd doesn't all knock at the same second. After a few tries with no answer, they give up and leave a note instead of standing there forever. That pattern of waiting longer each time, with a random nudge, is exactly how a good retry policy treats a flaky API.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Calling an LLM API is not like calling a fast, deterministic internal service. Providers throttle you with 429s, return transient 5xxs under load, and — most dangerously — sometimes just stall mid-decode without ever returning. A naive client that fires the same request again the instant it fails turns each of these blips into a self-inflicted incident: it stacks load onto an already-struggling dependency and synchronizes every client into one giant spike.
The fix is a retry policy whose steps run in a specific order, where each step exists to neutralize a specific failure mode. Reorder them and the policy quietly stops protecting you. Put the fallback before the cap and the loop never terminates. Drop the timeout and the whole policy can't run because the original call never returns.
This deep dive walks the six rungs in order — timeout, classify, backoff with jitter, retry, cap, fall back — explaining the reasoning behind each placement, the math that makes backoff and jitter work, and the production failure modes that show up when a team gets the order wrong.
Why the timeout has to be the very first rung
The most common way an LLM call hurts you isn't an error code — it's silence. Under heavy load a provider can accept your request, begin streaming, then stall. The socket stays open, the worker stays blocked, and nothing in your retry logic ever fires because the call hasn't returned to be retried.
That's why the per-request timeout wraps everything else. It's the only mechanism that bounds an otherwise unbounded wait. Set it, and a hung call becomes a timeout exception you can catch, classify, and act on. Skip it, and a single slow request can pin a worker thread; a handful of them starve your connection pool and your service goes down for reasons that have nothing to do with your own code.
A subtle point: the timeout should be per-attempt, not for the whole retry sequence. Each individual call gets its own clock. You may also want a separate overall deadline so the total time spent across all retries stays bounded — useful when a user is waiting on the other end. But the per-attempt timeout is the non-negotiable first rung, because without it every later step is unreachable.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import random, time
def call_with_retry(fn, base=0.5, cap=8.0, max_attempts=4, timeout=20):
for attempt in range(max_attempts):
try:
return fn(timeout=timeout) # 1. per-request timeout
except Exception as e:
if not is_retryable(e): # 2. classify
raise
if attempt == max_attempts - 1: # 5. attempt cap
break
delay = min(cap, base * 2 ** attempt) # 3. exponential backoff
delay *= 1 + random.random() # + jitter
time.sleep(delay) # 4. wait, then loop retries
return degraded_fallback() # 6. graceful degradationReal products, models, and research that use this idea.
- AWS SDKs ship full-jitter exponential backoff by default — the canonical reference for de-synchronizing retries.
- LLM gateways like LiteLLM and Portkey implement timeout, retry with backoff, and fallback routing as built-in policies.
What an interviewer would ask next. Try answering before peeking at the approach.
QCompare full jitter, equal jitter, and decorrelated jitter. When would you pick each?
Full jitter samples uniformly in [0, backoff] and spreads clients best; equal jitter keeps half the delay fixed for more predictability; decorrelated jitter grows from the previous delay to avoid collapsing to zero.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Retrying instantly in a tight loop with no timeout, no error classification, and no attempt cap — which turns one slow provider into a self-inflicted outage.
60 second bullets to scan on the way to the call.
Why the per-request timeout has to come before any retry logic
Which HTTP status codes are retryable versus deterministic failures
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.