- 1Hold blue on standby for the agreed rollback window, then tear it down
- 2Mirror a sample of production traffic to green and diff quality, latency, and cost against blue
- 3Ramp green to 100 percent while blue stays warm
- 4Shift a small percentage of live traffic to green with guardrail metrics and automatic rollback wired in
- 5Stand up the green stack on the new model, configured identically but taking zero user traffic
- 6Pin the candidate model ID and run the offline eval suite against the current champion
Move from cheap evidence (offline evals) to expensive evidence (live traffic) in steps, keep blue warm as your rollback, and never tear it down until the window closes.
Imagine swapping the chef at a busy restaurant. You do not just shove the new chef into the kitchen on a Friday night. First you taste their food in a private trial. Then you let them cook in a copy of the kitchen with no real diners. Then you let them quietly cook the same orders the old chef is cooking, and compare plates side by side. Only then do you send a few real tables their way, watching for complaints. If everything looks great, more tables go their way until the whole dining room is theirs. The old chef stays in the building, apron on, for a while longer just in case the new one has a bad night.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Model swaps look simple on a slide: point the gateway at a new ID and let traffic flow. In practice, that one-step swap is the single most common cause of self-inflicted LLM incidents. The candidate model differs from the champion on tokenizer behaviour, safety filtering, rate limits, and latency shape in ways that no eval set catches in full, and the moment real traffic hits, the failure surface is enormous.
The blue-green pattern, adapted from web service deployment, exists to control that surface. You build a parallel stack (green), gather increasingly costly evidence that it behaves at least as well as the live stack (blue), and shift traffic in measured steps with a fast path back to blue at every stage. The order of those steps is not aesthetic; it is the cheap to expensive ordering of the evidence you can collect.
This deep dive walks the six stages, names what each one uniquely catches, and closes with the two invariants that turn a swap from heroic to boring: warm blue and automatic rollback.
Stages one and two: pin, eval, and stand up green
Pinning the candidate model ID is the unglamorous first move and the one that most often gets skipped. Models behind an alias like claude-sonnet-latest move under your feet; a swap test against a moving target is not a swap test. Pin to a full versioned ID for both blue and the candidate, and the rest of the process becomes reproducible.
With both IDs pinned, the offline eval suite runs against both with their respective adapted prompts. Prompts are not portable between models: a system prompt tuned over months for the champion will sandbag a challenger that responds differently to certain phrasings. Evaluate each model with its own prompt variant and version both alongside the model ID. Offline catches blatant capability regressions for almost no money and zero user risk, which is exactly the property that earns it the first slot.
Standing up green next is deliberately not a quality step. It is an infrastructure shakedown: SDK versions installed, secrets propagated, new parameter names (max_completion_tokens, region-specific endpoints, refreshed safety-filter knobs) wired correctly, capacity reserved. If you discover during canary that green cannot accept a parameter the champion was using, you have wasted everyone's time. Stand it up, smoke-test it with synthetic requests, watch for cold-start surprises, and only then expose it to anything resembling real traffic.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LiteLLM and OpenRouter gateways let teams pin model IDs per route and shift weighted traffic between them, which is the practical knob behind the canary and ramp steps.
- Helicone and Langfuse traces are commonly used as the mirror-diff layer: tag requests with a model variant and the dashboards surface side by side latency and cost.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you size the canary percentage and duration?
Size from the metric you want to detect: pick the smallest regression you care about, then compute the traffic and duration needed for that effect to clear a chosen significance threshold.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the swap as a single cutover step, or tearing down blue the moment green hits 100 percent and discovering a regression with no fast rollback path.
60 second bullets to scan on the way to the call.
Why does offline evidence come before any infrastructure work?
What failure mode does standing up the green stack with zero traffic uniquely catch?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.