Canary ramps for prompt rollouts go 5 percent then 25 to 50 percent then 100 percent, with the first soak window lasting 30 minutes or so to surface judge-score regressions.
Think of opening a new restaurant kitchen by serving only a few tables first. If the food comes out fine for those five tables, you open it to a quarter of the room. If that still goes well, the whole room gets the new kitchen. The whole point is that if something is wrong, only a few people get the bad dish before you catch it. Prompt rollouts work the same way. Send a small slice of users to the new prompt, watch the dashboards for half an hour or so, then ramp up if nothing looks off. The numbers (5 percent, then 25, then 100) come from years of teams getting burned by all or nothing flips.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Canary rollouts moved from web infrastructure into LLMOps almost unchanged in form, with a few new twists driven by the cost and statistical properties of LLM traffic. The basic shape (5 percent, then 25 to 50, then 100, with a soak at each stage) is the same one a backend team would use for a risky API change. What is different is the signal being measured (judge scores, refusal rates, per-tenant deltas) and the cost calculus (every canary request costs real money on a provider bill).
This walkthrough covers the canonical ramp, why each number is what it is, how to size both the percentages and the soak windows to your traffic, and the failure modes that bite teams who treat the numbers as gospel instead of as a starting heuristic.
One-line summary: the percentages are the easy part; the soak duration is where teams under-invest and where regressions slip through.
The canonical ramp and where the numbers come from
The standard three stages
A typical prompt-rollout canary runs:
- 5 percent of traffic for a 30 to 60 minute soak.
- 25 to 50 percent for a shorter soak (often 15 to 30 minutes).
- 100 percent cutover, with the old prompt kept warm for instant rollback.
Some teams start at 1 percent for very high traffic services where 5 percent represents thousands of requests per minute. Some start at 10 percent for low-volume services where 5 percent is too small to produce a statistically usable sample.
Why 5 percent and not 1 or 10
5 percent is the sweet spot for most production traffic: small enough that a bad prompt damages a contained slice, big enough that judge scores aggregate to a usable signal within an hour. At 1 percent the signal to noise is poor and you wait too long; at 10 percent the blast radius is high enough that a hard regression hurts.
Why a middle stage
Going 5 to 100 skips the confidence-building step. The middle stage is your chance to verify that the metric stability you saw at 5 percent holds at meaningfully larger traffic; sometimes regressions only appear under load because of caching, rate-limit headroom, or downstream service interactions.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Statsig and LaunchDarkly both ship prompt-rollout templates with 5/25/100 as the default ramp curve.
- LangSmith and Langfuse integrate with feature-flag platforms to compute parallel judge-score deltas during a canary soak.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you size the first canary slice for a service doing 50 requests per minute?
You need roughly 500 to 1000 graded responses in the soak window. At 50 RPM, 5 percent is 2.5 RPM, which needs 4 to 7 hours; that is too slow. Either start at 20 percent (10 RPM, soak in 1 to 2 hours) or batch the canary with offline evals during a longer window.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping the soak window. A 5 percent slice for five minutes is not a canary, it is a slow rollout. Real regressions need 20 to 60 minutes of traffic to surface in judge-score aggregates.
60 second bullets to scan on the way to the call.
The canonical 5/25/100 ramp curve
Soak duration sized to traffic volume and signal variance
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.