- 1Full rollout to 100 percent with rollback flag retained
- 2Internal dogfood: new prompt served to employees only via feature flag
- 3Canary ramp to 25 then 50 percent with watch windows
- 4Canary 1-5 percent traffic with sticky per user assignment
- 5Shadow mode: new prompt runs in parallel with old, only old response served to user
- 6Eval CI golden set replay on the PR (offline, no production traffic)
Eval-CI then shadow then dogfood then small canary then ramp then full rollout, keeping the rollback flag live even at 100 percent so reverting a regression is a flag flip not a deploy.
Think of opening a new recipe at a restaurant. First you cook it once in the test kitchen with no customers. Then you cook it alongside the old recipe but only serve the old one to guests so you can taste-compare without risking dinner. Then you feed it to the staff to catch obvious problems. Then you serve it to a few tables with their permission, then to a quarter of the dining room, and finally to everyone. You keep the old recipe printed and ready the whole time, so if something tastes off at any point you can switch back in seconds without closing the kitchen. Prompt deployments work the same way: each step exposes the change to more people, and the rollback path stays warm to the end.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt deployments are behavior changes to production services, and the discipline borrowed from web operations applies in adapted form. The risk profile is different (LLM regressions are often subtle, cohort-specific, and hard to detect from aggregate metrics) which makes the controlled-rollout ladder more important, not less. The six-step ladder is the 2026 standard pattern in serious LLMOps teams.
This card walks through what each step contributes, why the order matters, and how to keep the rollback path warm through full rollout.
The first three gates: eval-CI, shadow, dogfood
The first three steps gate the change with progressively more realistic signal and zero or near-zero user impact.
Eval-CI runs the new prompt against a versioned golden set on every PR. The set is 200-500 prompts covering representative production patterns, each with a judge-scored reference for faithfulness, task-success, refusal-rate, and format-validity. The eval is a gate: a regression blocks merge unless explicitly accepted with sign-off. The set grows over time as new failure classes are discovered through incidents.
Shadow mode runs the new prompt in parallel with the old on real production traffic but returns only the old response. The shadow responses are logged and batch-judged offline. This catches behavior shifts that the golden set could not anticipate because the production traffic distribution differs. Long-tail prompt shapes, rare tenant patterns, and unusual languages all show up here. Run shadow long enough to accumulate statistical power on the metrics you care about (hours for high-volume surfaces, days for low-volume).
Dogfood routes employee traffic to the new prompt via a feature flag. Humans review real outputs in a low-stakes population. This catches tone, style, refusal-pattern, and verbosity issues that score fine on a judge model but feel wrong to a human reader. It is the cheapest qualitative review step in the ladder.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's internal prompt-deploy pipeline keeps shadow runs for hours before any user-facing change.
- OpenAI's model and prompt rollouts use sticky per-user canaries via LaunchDarkly-style flag systems with cohort metric watches at each bump.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow long should shadow mode actually run?
Long enough to accumulate statistical power on the metrics you gate on; depends on traffic volume, metric variance, and the size of the regression you care about detecting.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Removing the rollback flag the moment rollout hits 100 percent. Keeping it costs nothing and saves the next incident response.
60 second bullets to scan on the way to the call.
Eval-CI golden-set as the offline gate
Shadow mode purpose: catch distribution shifts the golden set missed
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.