Why a circuit breaker around the primary LLM provider is more than a fancy retry

A **retry** says, this single request failed, try again. A **circuit breaker** says, this dependency is sustained-broken, stop calling it entirely for a while. The retry is the right tool for transient blips: a single 503, a one-off rate-limit response, a network hiccup. Three retries with exponential backoff and you usually succeed. The circuit breaker is the right tool when the failure is not transient. The breaker tracks failure rate over a rolling window. When it crosses a threshold (say, 50 percent failure over the last 30 seconds), the breaker opens. While open, every new request short-circuits straight to the fallback path without even attempting the primary provider. The breaker periodically transitions to a half-open state: it lets a small slice of traffic through to test the primary. If those succeed, it closes (back to normal). If they fail, it reopens for another cooldown window. This matters for LLM providers because outages tend to be long. A 20-minute incident with naive retries means every user request waits the full retry budget before falling back, p99 latency explodes, and you spend rate-limit budget hammering a dependency you already know is down.

Retries handle one-off failures; **circuit breakers handle sustained ones**. The two compose: retries on the primary inside the breaker, and when the breaker trips, requests short-circuit to the fallback path with no primary attempt at all. During a sustained provider outage, retries actively make things worse. Every request waits the full retry budget (say, 3 attempts with backoff = 10 to 30 seconds) before falling back, so user-perceived p99 inherits that budget across the entire incident. Trace cardinality explodes because every failed call emits a span. And on the recovery side, the provider's rate-limiter sees your hammering and starts shedding load right when capacity returns. The breaker collapses that to one fast fallback decision per request. While open, fallback latency is the only latency. The half-open probe gives a small percentage of traffic to the primary to detect recovery without committing the whole fleet. Implement either inside a gateway (Portkey, LiteLLM proxy, custom envoy filter) so the policy is shared across services, or in application code with a library like Resilience4j (JVM) or PyBreaker (Python). Gateway placement is the production default; one config knob, no client redeploy needed when tuning thresholds.