with_fallbacks is ordered failover. On exception, it advances to the next backup Runnable. It never retries the same one, never runs concurrently and never silently swallows errors.
Imagine you have three friends to ask for a ride home. You call the first. If she does not answer, you call the second. If he does not answer, you call the third. You do not redial the first friend over and over, you do not call all three at once, and if all three fail to answer you give up and figure out a Plan D. That is how with_fallbacks works: a list of friends, called in order, with no redialing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Production LLM stacks depend on their reliability primitives. Provider outages, rate limits, and transient network errors are routine. An application without a failover plan goes down whenever its primary provider has an incident. LangChain's two reliability primitives, with_fallbacks and with_retry, address two distinct failure modes. Production bugs tend to appear where they are confused.
Three points matter: failover is not retry, fallback calls are sequential rather than concurrent, and when every fallback fails the call raises rather than silently swallowing errors. Each wrong option maps to a misconception that produces a different class of production bug. 'Retry semantics' is the most common mistake. 'Concurrent semantics' produces wrong cost expectations. 'Silent swallowing' produces hidden outages that nobody pages on.
What with_fallbacks does
with_fallbacks wraps a Runnable with an ordered list of backup Runnables. The wrapped object is itself a Runnable. Calling it invokes the primary first. If the primary raises an exception (filtered by exceptions_to_handle, defaulting to Exception), the wrapper invokes the first backup with the same input. If that backup also raises, the wrapper invokes the second backup. The pattern continues through the list. If every backup fails, the wrapper raises the first error, the one from the primary.
The shape is primary.with_fallbacks([backup1, backup2, backup3]) with optional keyword arguments: exceptions_to_handle=(SomeException,) scopes which exceptions trigger failover, and exception_key=None controls whether the handled exception is passed to the backup (under that key in a dict input).
The mental model. Think of with_fallbacks as 'try a different thing.' The primary is one Runnable; each backup is a different Runnable. They might call different models, different providers, different prompts, different chains. The only constraint is that they accept the same input shape and produce the same output shape. The calling code then does not know which one served the request.
The call tree is linear. Primary, then (on failure) backup1, then (on failure) backup2. The total latency on a multi-backup failure is the sum of all attempted calls' latencies. A primary that times out at 30 seconds followed by a backup that takes 2 seconds produces a 32-second total. This is usually fine because the failure path is rare. But it matters for SLO calculations: the p99 latency of a with_fallbacks chain is the worst-case path through every backup, not the median path.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Provider failover patterns: primary ChatOpenAI with gpt-5.5, fallback to ChatAnthropic with Claude Opus 4.7. This is the most common use case.
- Cost-tier fallback: premium model as primary, cheaper model as backup; if the premium is rate-limited, the cheaper model serves the request.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through the composition pattern for retry plus fallback in detail.
primary = ChatOpenAI(...).with_retry(stop_after_attempt=3, exponential_jitter_params={"initial": 1, "max": 10}). backup = ChatAnthropic(...).with_retry(stop_after_attempt=2). chain = primary.with_fallbacks([backup], exceptions_to_handle=(APIError, RateLimitError)). Each Runnable retries its own transient errors; only if a Runnable exhausts its retries does the fallback chain advance.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing failover (next Runnable on failure) with retry (same Runnable again). with_fallbacks is the first; with_retry is the second. Mixing them up causes thundering retries on the wrong call.
60 second bullets to scan on the way to the call.
What the with-fallbacks wrapper does (ordered failover, sequential, raises on all-fail)
What the with-retry wrapper does and how it differs from fallbacks
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.