The three states of a circuit breaker around a model provider
A circuit breaker stops hammering a failing model provider: CLOSED counts failures normally, OPEN fails fast after the threshold trips, and HALF-OPEN sends probe requests to test recovery before fully reopening.
Imagine an electrical fuse in your house. When something shorts out, the fuse trips and cuts the power so the wiring doesn't catch fire. You don't keep flipping the switch back on every second — you wait, then test it once to see if the problem's gone. A software circuit breaker works the same way around a model provider. When the provider keeps failing, the breaker 'trips' and stops sending requests for a while, so you don't waste effort and money on a dead service. After a pause, it cautiously sends one test request to check if things recovered before turning the flow back on.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The circuit breaker takes its name and its logic straight from household electrical wiring. When a circuit draws too much current, a fuse or breaker trips and cuts the power, protecting the wiring from overheating. You don't keep jamming the switch back on; you wait, fix the problem, and reset. The software pattern transplants that idea to a service calling a flaky dependency.
In an LLM system, the dependency is a model provider, and the thing the breaker protects is your own service's capacity. The non-obvious threat isn't a provider that returns clean errors — those are easy. It's a provider that gets slow, leaving your requests hanging near their timeout, each one holding a thread until your service runs out and falls over too.
This deep dive explains the cascading-failure problem the breaker solves, walks the CLOSED, OPEN, and HALF-OPEN states and why three is the right number, contrasts the breaker with retries (a pairing people constantly confuse), and shows how the OPEN state becomes the trigger for graceful degradation in a real LLM stack.
The cascading failure a breaker is built to stop
To see why the breaker exists, picture your service under normal load with a healthy model provider. Requests come in, you call the provider, responses come back in a few hundred milliseconds, and threads free up quickly. Everything flows.
Now the provider degrades — not failing cleanly, just slowing down. Calls that took 300ms now take 25 seconds and hang near your timeout. Here's the trap: every hung call holds a worker thread or a connection from your pool for the entire timeout. New requests keep arriving, each grabs a thread, each thread waits 25 seconds. Within minutes every thread is stuck waiting on the sick provider, and your service can't accept new work even though your code is fine.
That's a cascading failure: a problem in a downstream dependency propagates upstream and takes you down too. And retries make it strictly worse, because they add even more calls to the struggling provider.
The breaker's whole purpose is to break this chain. When it detects the provider is unhealthy, it stops sending requests there and fails them fast instead. Fast failures don't hold threads. Your service stays responsive — it returns a fallback or an honest error in milliseconds — rather than slowly suffocating on hung calls. The breaker trades a degraded experience for survival of your own service.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Conceptual circuit-breaker state machine
if state == "OPEN":
if now - opened_at > cooldown:
state = "HALF_OPEN" # time to probe recovery
else:
return fallback() # fail fast, don't call provider
if state in ("CLOSED", "HALF_OPEN"):
try:
result = call_provider()
if state == "HALF_OPEN":
state = "CLOSED" # probe succeeded -> reopen flow
failures = 0
return result
except ProviderError:
failures += 1
if state == "HALF_OPEN" or failures >= threshold:
state, opened_at = "OPEN", now # trip the breaker
return fallback()Real products, models, and research that use this idea.
- LLM gateways like LiteLLM and Portkey implement per-provider circuit breakers that trip and route to a fallback model.
- Resilience libraries Resilience4j (JVM) and Polly (.NET) are the canonical circuit-breaker implementations teams reuse.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you set the failure threshold and cooldown so the breaker neither flaps nor stays open too long?
Use a rolling window with a minimum request volume before tripping, an error-rate threshold rather than a raw count, and a cooldown long enough for recovery but short enough to retest soon.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing a circuit breaker with a retry loop — retries keep calling a failing provider, while the breaker's whole job is to stop calling it until it recovers.
60 second bullets to scan on the way to the call.
What failure mode a circuit breaker prevents that retries don't
What happens in the CLOSED state and what it tracks
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.