- 1Retry the primary model once with backoff (the failure may be transient)
- 2Surface a hard error only if every prior step is impossible
- 3Serve a cached or pre-computed answer if one exists for this query
- 4Route to a cheaper or faster secondary model
- 5Return a templated, honest 'we're busy: try again shortly' response
Walk a fallback ladder from highest-fidelity to lowest: retry the primary, route to a cheaper model, serve a cache, return an honest wait message, and only then surface a hard error.
Imagine your favorite restaurant is slammed and the head chef can't cook your dish right now. A good waiter doesn't just slam the door in your face. First they check if the chef will be free in a minute. If not, they offer a simpler dish another cook can make. If even that's impossible, they bring out something already prepared earlier. And if the whole kitchen is on fire, they at least say 'we're swamped, please come back soon' politely — they never just walk away silently. That ladder of fallbacks, each a bit less ideal but never a slammed door, is graceful degradation.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
When a chat app's primary model gets overloaded, the lazy design returns an error and moves on. That's the worst possible outcome: the user came with a task, hit a blank wall, and many of them simply leave. Graceful degradation is the discipline of never showing that blank wall when a worse but still useful option exists.
The core idea is a ladder of fallbacks, ordered by how good the resulting experience is. Each rung is a deliberate trade — a little quality surrendered to keep availability high. The interview skill being tested is whether you can rank those rungs correctly, because the ordering is not arbitrary: it follows directly from how good the answer is that each rung can still produce.
This deep dive establishes the ranking principle, walks each rung and why it sits where it does, examines the one rung people consistently misplace, and covers the operational half that's easy to forget — that degrading quietly for users only works if you alert loudly for engineers at the same time.
The ranking principle behind the ladder
The ladder isn't memorized; it's derived. Rank every fallback option by a single question: how good is the answer the user ends up with? Then order them best to worst, and you have the ladder.
The top of the ranking is a fresh answer from the full-quality primary model. That's what you'd serve if nothing were wrong, so anything that can recover it — a quick retry — sits at the top. Below that comes a fresh answer from a lesser model: still responsive to the exact query, just lower quality. Below that comes a real but stale answer from cache. Below that, no answer at all but an honest message. At the very bottom, a hard failure.
The key insight is the tiebreaker between 'fresh but lower quality' and 'high quality but stale'. Freshness usually wins, because a live model actually answers this user's current question, while a cache can only return what someone asked before. A slightly weaker fresh answer beats a perfect answer to a different question.
Once you internalize this ranking, you don't memorize five steps — you reconstruct them from first principles, and you can defend any ordering challenge an interviewer raises.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LLM gateways like LiteLLM and Portkey implement fallback model chains that auto-route to a secondary provider on primary failure.
- Cloudflare AI Gateway and similar proxies support cached-response fallback when an upstream model is unavailable.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you decide which queries are safe to serve from cache during degradation?
Cache works for idempotent, non-personalized, or slowly-changing queries; never serve a cached answer for a personalized or time-sensitive request where staleness would mislead the user.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Jumping straight to a generic error or a cached answer instead of first trying the cheaper live model that could still produce a fresh, correct response.
60 second bullets to scan on the way to the call.
Why a retry with backoff belongs at the top of the ladder
How to rank a live cheaper model against a cached answer
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.