Match top-1 vs top-2 MoE routing to their tradeoffs
Drag each answer to line up with its matching prompt
Top-1 (Switch Transformer)
Roughly 2× expert FFN compute with better quality and routing redundancy
Top-2 (Mixtral style)
Minimal expert FFN compute per token but higher routing collapse risk
Top-1 serving
Simpler dispatch, each token hits one expert GPU path
Top-2 serving
More all to all traffic because each token may activate two expert shards
Top-1 minimizes expert FFN compute and dispatch fanout at higher collapse risk; top-2 doubles both for quality redundancy.
Picture two delivery routes: one truck per package (top-1) is cheaper but risky if that truck breaks down; two trucks per package (top-2) costs more but has backup. Match training pairs to compute vs quality, and serving pairs to one hop vs two hops.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Match_pairs questions reward structured knowledge, you should see two clusters in the four rows: training/architecture tradeoffs and serving/systems tradeoffs. Both clusters scale with k, but they measure different resources (FLOPs vs network).
Candidates who memorize "Switch=1, Mixtral=2" without the serving row often swap dispatch pairs. This deep dive locks both dimensions.
Use landmark model names as anchors while explaining mechanism, interviewers accept shorthand if the tradeoff logic is precise.
Match_pairs questions reward structured knowledge, you should see two clusters in the four rows: training/architecture tradeoffs and serving/systems tradeoffs. Both clusters scale with k, but they measure different resources. Candidates who memorize Switch=1 and Mixtral=2 without the serving row often swap dispatch pairs.
The four-pair grid is symmetric: two training rows, two serving rows. Interviewers who ask match_pairs often follow with "which row would you optimize first in production?", usually serving row 4 (communication) or training row 1 (collapse).
Before matching, sort the four right-side descriptions into two buckets: training claims (FLOPs, collapse, quality) and serving claims (dispatch hops, all to all volume). Mixing buckets is the primary error mode.
Senior interviewers reward candidates who connect this mechanism to a concrete deployment or training decision, not only the textbook definition. Close with one number, one failure mode, and one monitoring signal you would track in production.
Pair 1: Top-1 compute vs collapse
Switch Transformer anchor. Top-1 activates one expert FFN per token, minimum conditional compute. The router's decision is all or nothing: one expert receives the full FFN transform responsibility.
That sharp selection raises routing-collapse risk: popular experts attract more tokens → receive more gradient → become more popular. Aux loss and capacity factor mitigate but do not eliminate skew.
Right side of pair 1 names both benefits (minimal FFN compute) and cost (higher collapse risk), answer both halves.
Training-side pair 1. Top-1 activates one expert FFN per token, minimum conditional compute. The router's decision is all or nothing for FFN transform responsibility. Sharp selection raises routing-collapse risk because the popular expert attracts more tokens, receives more gradient, and becomes more popular. Aux loss and capacity factor mitigate but do not eliminate skew. Right side names both benefit (minimal FFN compute) and cost (higher collapse risk).
Gradient flow detail: top-1 sends 100% of FFN gradient through one expert per token. Top-2 splits gradient across two paths, reducing per-expert gradient magnitude and slowing collapse feedback.
Pair 1 left side is top-1; right side must mention both low FFN compute AND collapse risk. If your match only mentions compute, it is incomplete.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | Top-1 routing | Top-2 routing |
|---|---|---|
| Experts active per token | One FFN path | Two FFN paths blended |
| Collapse risk | Higher winner take all pressure | Lower, second expert shares load |
| Active FLOPs per token | Lower | Roughly 2× top-1 expert FFN |
| Dispatch volume | One expert destination | Two destinations, more all to all traffic |
| Typical lineage | Switch Transformer | Mixtral 8×7B |
Real products, models, and research that use this idea.
- Switch Transformer training runs target top-1 + capacity factor for trillion-scale FLOP efficiency.
- Mixtral serving on vLLM dispatches each token to two expert ranks per MoE layer.
What an interviewer would ask next. Try answering before peeking at the approach.
QDoes top-2 always mean exactly 2× expert FFN FLOPs if gate weights zero one expert?
Both experts still run after top-k selection; renormalization affects weights, not whether two FFNs execute.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Swapping serving pairs, top-2 has more all to all traffic, not simpler dispatch.
60 second bullets to scan on the way to the call.
Switch equals top-1 landmark
Mixtral equals top-2 landmark
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.