Order the rollout steps for migrating a dense serving stack to an 8-expert MoE.

Order steps·Medium·4.0 · 0·~1 min·Asked atCanva Stability Ai Writer

Attempt it

1Complete the cutover with an automated rollback trigger tied to quality and latency SLOs
2Canary a small user slice with per expert utilization and token drop dashboards watching for skew
3Size VRAM for all resident experts and choose quantization plus the expert placement/parallelism layout
4Load test the MoE stack, measuring all to all dispatch overhead and per expert load under peak traffic
5Shadow deploy the MoE beside the dense model, comparing answer quality and tail latency on mirrored live traffic
6Profile representative production traffic and record the dense baseline's quality and latency targets

**Migrating from dense to MoE serving** is a six-stage rollout where the order matters more than people expect. The pattern is: know your baseline, design the system, stress-test offline, then expose real traffic in increasing slices. First, **baseline the dense stack**: production quality scores, p50 and p99 latency, throughput per GPU, and the SLOs the product promises. You need numbers to beat, not vibes. Second, **design capacity and parallelism**: VRAM for all experts (quantisation choice), expert-parallel placement across GPUs, and the dispatch path. This is where MoE-specific decisions live. Third, **load-test offline**: synthetic peak traffic that probes all to all dispatch overhead and per-expert load. Catch dispatch bottlenecks before any user sees them. Fourth, **shadow-deploy**: mirror live requests to the MoE stack without serving its responses. Compare quality and tail latency offline. Fifth, **canary**: route a small slice of real users to MoE. Watch per-expert utilisation and token-drop dashboards for skew. Sixth, **cutover** with an automated rollback tied to quality and latency SLOs.

**The right order is baseline, design, synthetic load, shadow, canary, cutover.** Skipping or reordering steps is how MoE migrations turn into incident-review case studies. **Baseline first** because you cannot evaluate a migration without numbers from the system it replaces. Capture quality scores on your eval suite, p50 / p99 / p999 latency, throughput per GPU, error rates, and the product's actual SLOs. This is also the moment to record cost per million tokens so you can compare honestly later. **Capacity and parallelism design** comes next. MoE memory is dominated by total expert weights, not active compute, so you size VRAM for full residency. Pick quantisation (fp8 or int4 for experts is standard in 2026), expert-parallel layout (one or two experts per GPU is typical for 8-expert configs), and the all to all dispatch path. **Synthetic load test** before any real traffic. Drive peak QPS and measure all to all overhead per layer, per-expert load distribution under your actual traffic mix, capacity-factor adequacy, and tail latency. Routing skew that looks fine in averages can pin a single GPU under peak load. **Shadow deployment** mirrors live requests to the MoE stack without returning its responses. Offline diff the quality and latency against the dense baseline on real traffic distributions, not synthetic ones. **Canary** exposes a small slice of real users, with per-expert utilisation, token-drop counters, and quality regression dashboards watching live. Hold long enough to see your traffic's diurnal pattern. **Cutover** flips traffic with an automated rollback trigger wired to SLO breaches, not human judgment in the middle of an incident.

TL;DR

Baseline first, then capacity and parallelism design, then synthetic load tests, then mirrored shadow, then canary, then SLO-gated cutover. Real-traffic exposure escalates in stages.

Memory aid

Easy to grasp

Think about moving a kitchen from one cook to a team of specialist chefs. First, write down how fast the old cook is and what the customers like, so you have something to beat. Then design the new kitchen: where each chef stands, what tools they share, how orders get passed around. Run a fake dinner service with pretend customers to see if anything explodes. Next, mirror real orders to the new kitchen but do not actually serve those plates yet. When that goes well, send a few real customers to the new kitchen while everyone else stays with the old cook. Finally, switch everyone over, but keep a button that sends them back instantly if the new kitchen falters.

Key concepts

Mixture of Experts Inference Optimization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Migration rollouts are about ordering, not just contents. Most teams know all six stages exist. Fewer can articulate why they must happen in this specific order, and the reason becomes obvious only when a rollout goes wrong: each stage's output is a precondition for the next stage's decisions, and skipping ahead means making decisions without the data that justifies them.

This walkthrough explains the dependency chain that locks in the order, the failure modes each stage uniquely catches, and where the common compression temptations actually save time versus where they hide risk that surfaces in production.

Mental model: the rollout escalates real-traffic exposure in stages, and each stage exists to detect failure modes the previous stage cannot.

Baseline before design (stages 1 to 2)

You cannot pick capacity, parallelism, or quantisation without knowing what the new stack must beat. Dense baseline numbers are inputs to the MoE design, not artefacts to compare against later. Latency SLOs decide whether you can tolerate all to all overhead; quality benchmarks decide whether quantisation aggression is acceptable; throughput targets decide expert-per-GPU layout.

Teams that skip this stage end up designing for hypothetical targets, then discovering during canary that the dense stack was actually faster at p99 because of routing variance. The baseline is also where you record cost per million tokens so the post-cutover finance comparison is honest rather than a vibes argument.

Stage two design uses baseline outputs directly. VRAM sizing accounts for total expert weights, KV cache for the measured concurrent user count, and all to all buffers for the chosen capacity factor. Quantisation choice (fp8 experts are standard in 2026) is informed by the quality regression budget set in stage one. Parallelism layout reflects your actual interconnect topology, not a textbook diagram.

Synthetic load before real traffic (stage 3)

Shadow then canary (stages 4 to 5)

Cutover with automated rollback (stage 6)

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Databricks DBRX migration playbooks document the baseline, shadow, canary, cutover pattern for moving Llama-class dense serving to DBRX-style MoE.
DeepSeek-V3 deployment notes emphasise per-expert utilisation dashboards and capacity-factor tuning before any traffic shift.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat dashboards do you build in stage two so canary in stage five has signals to watch?

Per-expert utilisation histogram, capacity-factor consumption, token-drop counter, all to all p50/p99, KV cache pressure. Wire them at design time, not after launch.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Jumping into canary or cutover before measuring all to all overhead and per-expert load under synthetic peak. Production traffic is the wrong place to discover dispatch is your bottleneck.

60 second bullets to scan on the way to the call.

Why baselining the dense stack must come before any MoE design decision
Which capacity and parallelism choices belong in stage two, not later

Primary sources. Browse if you want the original framing.