Spot the flaws in this three provider fallback chain

Spot the error·Hard·4.0 · 0·~2 min·Asked atForethought Labelbox Snap

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Fallback chains exist so that a single provider outage does not take down the product. This design has the right intent but five concrete flaws. First, it retries everything. A 400 error means the request was malformed; replaying it on two more providers cannot fix it. A content-policy block means the prompt violated a policy that every provider will likely also flag. The chain needs to classify errors and only retry the ones that retry can fix. Second, timeouts stack. Three providers at 60 seconds each means the user might wait three minutes before seeing an error. The total latency budget should be set at the user level and divided across providers, with each one getting less time than the previous. Third, the request is sent verbatim. Provider B has a smaller context window, and stop sequences are model-specific. Without adapting the request per provider, the failover lands on a guaranteed failure. Fourth, side effects from tool calls are not reversed. If a tool already sent an email, replaying creates a second email. Fifth, there is no circuit breaker. When Provider A is fully down, every single request still pays the 60-second wait.

**The chain has five orthogonal flaws and each maps to a standard reliability pattern.** *Error taxonomy.* Retrying everything wastes two more paid attempts on errors that cannot succeed anywhere. 400s are deterministic. Content-policy blocks port across providers. 401/403 are configuration bugs. Only 429s, 5xxs, and timeouts should fall through. Classify before failing over. *Latency budgets.* Stacking three 60-second timeouts means the user can wait three minutes. The right shape is one end to end budget (say 30 seconds at p99) divided across providers, with each subsequent attempt getting shorter timeouts so the budget holds. *Per-provider adaptation.* Prompts, stop sequences, model parameters, and context limits are not portable. Replaying a 128k prompt at Provider B with a 32k limit is a guaranteed failure. The chain must adapt the request per target: shrink the prompt, remap stop sequences, swap model parameters. *Idempotency for side effects.* A request whose tools already moved money or sent email cannot be replayed verbatim. Either use idempotency keys at the tool layer or persist execution state so the replay resumes rather than restarts. *Circuit breakers.* Without a breaker, a fully-down provider taxes every request with the full timeout. A per-provider breaker that trips on consecutive failures and skips the provider until a health probe recovers is standard.

TL;DR

The chain wastes cost on errors that retry cannot fix, stacks timeouts past any user patience, ignores per-provider request adaptation, double-fires side effects, and lacks circuit breakers.

Memory aid

Easy to grasp

Picture a relay race where the second runner cannot hear that the first runner fell. The whole team waits sixty seconds at the first baton drop before passing it. Then the second runner trips on a hurdle that only existed on their lane. Then the third runner sets off, and along the way some volunteers run out and hand prizes to spectators every time a runner passes, so by the end three sets of prizes have gone out for one race. That is what this fallback chain does. It waits, it does not adapt, and it re-fires side effects. A real chain skips known-down providers, bounds the user's total wait, reshapes the request per provider, and never re-executes work that already happened.

Key concepts

LLM System Design

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Multi-provider fallback is one of those patterns that looks trivial in a sprint plan and becomes a quiet liability in production. The trivial version is the try/except wrapper this v1 design embodies. It catches the right exception and retries on the next provider. It also wastes spend, blows the latency SLO, corrupts side-effecting tool calls, and tortures users during partial outages.

The five flaws in this snippet are not arbitrary. They map to five standard reliability patterns: error classification, latency budgeting, per-provider request translation, idempotency, and circuit breaking. Each is its own small subsystem. Together they turn the chain from a liability into something a senior engineer can defend in an incident review.

This deep dive walks each of the five and shows what a defensible chain looks like end to end.

Error taxonomy first, retry policy second

The hot path of any fallback chain is error classification. Retry policy without classification is just amplification of the wrong things.

Provider errors fall into three buckets. Deterministic errors are reproducible across providers given the same input: 400s for malformed requests, 401/403 for misconfigured auth, content-policy blocks for prompts that any major provider will reject, schema validation failures from structured-output mode. Retryable transient errors are correlated with provider state, not request state: 429 rate limits, 500/502/503/504 server errors, network timeouts that did not see the request completed. Ambiguous errors are the gray zone: timeouts mid-stream where some tokens may have been generated, partial tool calls.

The rule is simple: only retryable transient errors fall through to the next provider. Deterministic errors return immediately to the caller with the original status, since retrying cannot fix them and only burns money and time. Ambiguous errors need product-level policy: prefer to surface them to the caller rather than risk side-effect duplication.

One user budget, shrinking per-attempt allocation

Translate the request before retrying it

Idempotency for side effects

Circuit breakers and health probes

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LiteLLM and OpenRouter implement provider fallback with error classification and per-provider timeouts as first-class features
Vercel AI Gateway and AWS Bedrock cross-region routing combine circuit breakers with health probes to skip degraded endpoints

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen would you prefer hedged requests over sequential failover?

Hedging fires to two providers concurrently after a short head start delay and takes the first to return. It improves p99 latency at the cost of duplicate spend. Pick it when latency tail matters more than cost, typically for interactive surfaces with high cost per millisecond.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating every non-200 as retryable feels safe but means a 400 from a malformed prompt or a content-policy block costs three full provider round trips, three bills, and three times the user wait.

60 second bullets to scan on the way to the call.

How to classify provider errors into deterministic, retryable, and ambiguous
Why a user-facing latency budget beats stacked per-provider timeouts

Primary sources. Browse if you want the original framing.