- 1Retry the primary model once with backoff (the failure may be transient)
- 2Surface a hard error only if every prior step is impossible
- 3Serve a cached or pre-computed answer if one exists for this query
- 4Route to a cheaper or faster secondary model
- 5Return a templated, honest 'we're busy: try again shortly' response
Walk a fallback ladder from highest-fidelity to lowest: retry the primary, route to a cheaper model, serve a cache, return an honest wait message, and only then surface a hard error.
Imagine your favorite restaurant is very busy and the head chef can't cook your dish right now. A good waiter doesn't just turn you away. First they check if the chef will be free in a minute. If not, they offer a simpler dish another cook can make. If even that's impossible, they bring out something already prepared earlier. And if the whole kitchen has stopped working, they at least say 'we're swamped, please come back soon' politely. They never just walk away silently. That ladder of fallbacks, each a bit less ideal but never a flat refusal, is graceful degradation.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
When a chat app's primary model gets overloaded, the lazy design returns an error and moves on. That's the worst possible outcome. The user came with a task, got nothing back, and many users leave. Graceful degradation is the practice of never showing that empty error when a worse but still useful option exists.
The core idea is a ladder of fallbacks, ordered by how good the resulting experience is. Each rung is a deliberate trade: a little quality surrendered to keep availability high. The ordering is not arbitrary: it follows directly from how good the answer is that each rung can still produce.
Degrading quietly for users also only works if you alert loudly for engineers at the same time.
The ranking principle behind the ladder
The ladder isn't memorized. It's derived. Rank every fallback option by a single question: how good is the answer the user ends up with? Then order them best to worst, and you have the ladder.
The top of the ranking is a fresh answer from the full-quality primary model. That's what you'd serve if nothing were wrong, so anything that can recover it (a quick retry) sits at the top. Below that comes a fresh answer from a lesser model: still responsive to the exact query, just lower quality. Below that comes a real but stale answer from cache. Below that comes no answer at all, but an honest message. At the bottom is a hard failure.
The tiebreaker that matters is between 'fresh but lower quality' and 'high quality but stale'. Freshness usually wins. A live model answers this user's current question, while a cache can only return what someone asked before. A slightly weaker fresh answer beats a perfect answer to a different question.
Once you internalize this ranking, you don't memorize five steps. You reconstruct them from first principles and can defend any reordering.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LLM gateways like LiteLLM and Portkey implement fallback model chains that auto-route to a secondary provider on primary failure.
- Cloudflare AI Gateway and similar proxies support cached-response fallback when an upstream model is unavailable.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide which queries are safe to serve from cache during degradation?
Cache works for idempotent, non-personalized, or slowly-changing queries. Never serve a cached answer for a personalized or time-sensitive request where staleness would mislead the user.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Jumping straight to a generic error or a cached answer instead of first trying the cheaper live model that could still produce a fresh, correct response.
60 second bullets to scan on the way to the call.
Why a retry with backoff belongs at the top of the ladder
How to rank a live cheaper model against a cached answer
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.