Canary serves the new prompt to a small slice of real users (real signal, real blast radius); shadow runs the new prompt in parallel and never serves it (no blast radius, no engagement signal).
Picture a restaurant testing a new recipe. Canary is letting five out of every hundred guests order the new dish for real. You hear what they say, you see what they leave on the plate, and if the dish is bad, those five guests had a bad meal. Shadow is cooking both the old and the new dish for every order, serving the old one, and tasting the new one in the kitchen. Nobody at the table ever bites the new dish, so nobody complains. You learn how the new dish looks and smells, but you do not learn whether real diners would actually like it. Restaurants usually try the kitchen tasting first, then the small group of real diners, then the full menu change.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Canary and shadow are the two production-traffic test patterns every mature LLM team eventually adopts. The temptation is to use them interchangeably; both let you try a new prompt before committing. But they answer different questions and carry different costs, and the standard playbook uses them in sequence rather than as alternatives.
This walkthrough nails the exact mechanical difference (who sees the new output), maps each pattern to the kind of signal it produces, walks through the standard production sequencing, and flags the failure modes that bite teams who skip one step or the other.
Mental model: shadow is the kitchen tasting; canary is feeding a small group of real diners. The full menu change comes only after both pass.
The mechanical difference: who sees the new output
Canary serves the user
A canary rollout uses a feature flag (typically tied to user id for stickiness) to route a fraction of production requests to the new prompt. That fraction sees the new prompt's output as their actual response. The rest of traffic continues to see the old prompt.
Traffic percentages ramp through a standard ladder: 1 percent for an initial soak (hours to a day), then 5, 25, 50, and 100 if dashboards stay green. Auto-rollback at any step on regression.
Shadow mode does not serve the user
Shadow mode is a dual-write pattern. The application calls both prompts on the same input. The user response comes from the old prompt as usual. The new prompt's output is logged to the trace store (Langfuse, Phoenix, LangSmith) for offline comparison. The user never sees the new output.
Because no user sees it, there is no blast radius. A regression in the new prompt costs the team inference dollars and engineering attention, but zero user experience.
The contract this difference implies
Canary requires the new prompt to be production-ready: an actual user is about to read its output. Shadow tolerates a less polished new prompt because nobody will see it; shadow is where you find out whether the new prompt is production-ready. The sequence (shadow then canary) reflects this gradient of trust.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's own platform rolls Claude Opus 4.7 system-prompt updates through internal shadow eval before any tenant sees a change.
- LangSmith and Langfuse both support shadow evaluation hooks where production traces are dual-evaluated against a candidate prompt.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide between a 1 percent canary and a 5 percent canary for the same change?
Tie it to time to detect. The smaller the canary, the longer the soak needed to accumulate statistical signal on engagement metrics. For high-volume consumer traffic, 1 percent gives signal in hours; for B2B with low volume, you need 5 percent or higher just to see anything in a day.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping shadow and going straight to a 5 percent canary on a high-impact prompt. If the new prompt has a regression you did not catch in CI, 5 percent of users now hit it before any dashboard tells you.
60 second bullets to scan on the way to the call.
Definition of canary rollout and typical traffic percentages
Definition of shadow mode and what the user actually sees
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.