Order the safe rollout steps for migrating a single agent feature to a multi-agent team
- 1Ramp 25, 50, 100 percent on green metrics; keep the single agent path warm as fallback for at least one release cycle
- 2Add the kill switch feature flag and the per task cost ceiling before any production traffic hits the new team
- 3Canary the new team to 5 percent of users behind the flag, with auto rollback on quality or cost regression
- 4Shadow the new team in production on 1 percent of traffic (read only, output not served to users)
- 5Run the new team against the existing eval set and confirm it ties or beats the single agent on quality, with cost as a tracked secondary metric
- 6Build the multi-agent team in dev with the same tool surface as the existing single agent baseline
Build, beat the baseline on evals, install safety before any traffic, then shadow, canary, and ramp with the single-agent path warm as fallback.
Imagine swapping a working bus route for a fleet of three smaller buses that have to coordinate. You do not just switch the schedule overnight. You first build the three-bus version in a depot, test it on closed streets against the old route, install emergency brakes and a panic button, drive it empty along the real route to compare against the old one, then carry just a few volunteer passengers, then a few percent of riders, then more. The old bus stays warm in the garage the whole time in case you need to fall back. Multi-agent rollouts follow exactly the same caution pattern.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Migrating from a working single-agent feature to a multi-agent team is a high-leverage change with a high blast radius. Single-agent systems fail in ways operators have learned to predict. Multi-agent systems add cost explosions, talk loops, role bleed, and supervisor bottlenecks to the failure surface. The right rollout protocol absorbs those failures before users see them.
The order in this question is the standard production protocol: build, beat baseline, install safety, shadow, canary, ramp. Each step earns the right to the next. The senior signal is recognising that step 3 (safety) precedes any production traffic, because reversing that order makes the canary stage carry the risk it was supposed to absorb.
One-line summary: build, prove it on evals, brake first then drive empty, then with a few riders, then more. Keep the old bus warm.
Steps 1 and 2: build honestly and beat the baseline on evals
Same tool surface
The single-agent baseline has a specific tool surface. The multi-agent team should start with exactly that surface. If you add tools at the same time as you split into multiple agents, the comparison is confounded; any quality gain could be from the new tools rather than from the multi-agent decomposition. Lock the tool surface, decompose the agent logic only.
What 'beat the baseline' actually means
Run both systems against your golden eval set. The new team must tie or beat the single-agent on the headline metric (end to end success). Track per-agent step correctness, tool-call precision, cost per task, and latency as secondary metrics.
The failure mode this gate catches is the most common one: multi-agent is more expensive than single-agent by default, often 5 to 40x, and unless you apply the cost levers (cheaper supervisor, trimmed context, structured handoffs, hard caps) before shipping, you ship a regression on cost even when quality matches. The gate forces the cost conversation before any user is affected.
What 'cost as secondary' implies
Cost is tracked but it is not the gate. A 2x cost increase for a 10-point quality gain on a critical metric might be acceptable. A 10x cost increase for no quality gain is not. The eval data lets the team make that call honestly.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Klarna's customer-support agent rollout used shadow then canary with hard cost caps before ramping to full traffic.
- GitHub Copilot Workspace and Anthropic Claude Code both shipped multi-agent features behind feature flags with auto-rollback.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat specific metrics would you put on the auto-rollback trigger?
End to end success rate (drop more than X percent over the rolling window), per-task cost (rolling p95 exceeds the budget), latency (p95 exceeds SLA), complaint or thumbs-down rate (exceeds baseline by Y), and tool-call error rate (spikes above baseline). Tune thresholds against the canary period, not against eval numbers.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Adding the kill-switch after canary. Safety has to be in place before any production traffic, or the canary stage carries the risk it was supposed to absorb.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.