How does expert-choice routing guarantee load balance by construction?
Expert-choice caps each expert at top-m tokens per batch, which gives load balance by construction. But it needs batch visibility, which breaks causal streaming inference.
Think of a classroom roster. Instead of each student picking their favorite teachers (token-choice), each teacher picks their top 10 students from the whole class (expert-choice). No teacher gets more than 10. That is fair by design. But the roster must exist first, so it doesn't work when students arrive one at a time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Load balancing is a persistent problem in token-choice MoE. Expert-choice routing offers a direct fix: reverse the assignment direction and hard-cap each expert's intake at its top-m tokens. The guarantee is balance by construction. The price is batch visibility, which keeps expert-choice out of production causal LLM inference.
Token-choice overload problem
In token-choice MoE, each token independently selects top-k experts. Nothing prevents 70% of tokens from picking the same expert in a step, especially early in training or on narrow-domain fine-tunes. That expert becomes a straggler. Idle experts waste capacity.
The mitigations are auxiliary load-balancing loss (Switch f_i · p_i), DeepSeek dynamic bias, capacity-factor drops, and hash routing. All are soft or indirect: they steer or shed load but do not hard-cap per-expert intake in the token-choice paradigm.
Expert-choice asks: what if the expert, not the token, decides who it processes, up to a fixed budget?
Token-choice overload is a positive feedback loop: hot experts get more gradients → stronger on common patterns → router sends more traffic. Expert-choice interrupts the loop structurally by capping intake at m, with no waiting for aux-loss gradients to notice overload.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Expert-choice routing research demonstrates load balance without auxiliary loss on full-batch training steps.
- Mixtral and DeepSeek production stacks use token-choice routing for causal autoregressive inference.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat happens to cold experts under expert-choice?
They still select up to m tokens from their best-scoring candidates. The utilization floor improves vs token-choice collapse, where idle experts may get near-zero tokens.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Praising expert-choice load balance without naming the batch-visibility tradeoff that excludes causal LLM inference.
60 second bullets to scan on the way to the call.
Assignment direction inversion
Top-m cap per expert per step
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.