What should users see when the primary model is down, instead of an error page?
Your primary model provider has an outage. Describe how a well designed system degrades gracefully so users still get something useful, and why this beats returning errors.
Show users a degraded but useful response — a cheaper model, a cached answer, or an honest wait message — never a raw error, because a worse answer keeps the user's trust and task alive while an error page loses them.
Imagine you ask a librarian for a book and the main shelf is locked. A bad librarian shrugs and says 'system down, go away'. A good one checks a smaller branch, or hands you a photocopy they made earlier, or at least says warmly 'we're swamped right now, please pop back in five minutes'. Every one of those is better than being turned away at the door. The good librarian also quietly radios the manager that the main shelf is stuck, so it gets fixed. That mix — keep helping the visitor, and tell the boss something's wrong — is what a well-designed system does during an outage.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An outage at your model provider is not a question of if but when — every managed API has bad days, and rate limits or regional failures hit even the most reliable ones. The design question is what your users experience during those windows. The default, lazy answer is an error page. The good answer is that most users never realize anything was wrong.
Graceful degradation is the practice of always serving the best available outcome rather than failing hard. It rests on a simple but underappreciated asymmetry: a slightly worse answer costs you almost nothing, while a hard error can cost you the user permanently. That asymmetry is why building the fallback machinery is worth the effort.
This deep dive covers the user-facing ladder and the order its rungs take, the trust and completion argument for why degrading beats erroring, the operational discipline of alerting while you degrade, and the boundary cases where an honest failure is actually the correct, responsible choice rather than a cop-out.
What the user should actually see, rung by rung
Start from the user's seat. They asked a question and the primary model is down. Your job is to hand them the best outcome still possible, and 'best' has a clear ordering.
The top option is a fresh answer from a backup live model — a cheaper model, or one from an independent provider. It's responsive to their actual question, just possibly a notch lower in quality. Most users won't consciously notice the difference. Below that is a cached or pre-computed answer: real content, but from a past query, so it only fits when the question is a close repeat or slowly-changing.
When no answer can be produced, the user should see an honest, calm message — 'we're experiencing high demand, please try again in a moment' — rendered as ordinary UI. Not a spinner that hangs forever, not a 500, not JSON error text. The message tells them what happened and what to do, which keeps them oriented.
The through-line is that every rung is something useful or something honest. The user is never left staring at a broken page wondering whether the whole product is dead. That single guarantee — never a blank wall — is what graceful degradation delivers to the person on the other end.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- ChatGPT and Claude show a calm 'high demand, try again shortly' banner under load rather than raw errors.
- LLM gateways like LiteLLM and Portkey auto-route to a secondary provider when the primary model fails.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you keep a degraded experience from silently becoming the permanent normal state?
Alert on the rate of degraded responses and treat sustained degradation as an incident, so a fallback that should last seconds doesn't quietly run for hours.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating graceful degradation as purely a UX problem and forgetting the alerting half, so the system quietly serves degraded responses for hours with nobody on-call aware.
60 second bullets to scan on the way to the call.
The ordered fallbacks between a perfect answer and total failure
Why a live cheaper model outranks a cached better answer
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.