An 8-expert MoE layer trains with top-1 routing and no auxiliary load balancing loss. After 5k steps the per expert token fractions are: Expert 0 = 52%, Expert 1 = 31%, Experts 2–7 each under 3%. Expert 0 therefore receives roughly 17× more gradient updates per step than Expert 7. Training continues for another 10k steps with the same data distribution and no balancing regularizer. Calculate Expert 0's token fraction at step 15k.
Without load-balancing pressure, Expert 0's 52% share widens past 70% over 10k more steps via the routing-collapse positive feedback loop.
Imagine a tutoring center where one teacher already handles half the students and gets better every week. With no rule forcing fair distribution, more students pick the best tutor, who improves even faster. After months, one tutor handles most of the school.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The scenario gives concrete numbers: 52% leader share, 17× gradient ratio, top-1 routing, no aux loss and 10k more steps. The prediction needs both the direction and the magnitude of change. Routing share is not static. It co-evolves with expert quality through a positive feedback loop.
The prediction (>70%, typically 75–85%) is not a precise measurement. It is an illustrative range that follows from collapse dynamics.
The scenario removes every stabilizing mechanism: no aux loss, no capacity factor, no bias adjustment. So the reasoning starts from first principles: does the loop accelerate or stall? A training dashboard that shows one expert crossing 50% share by step 5k raises the same decision: intervene now or wait. A defensible range answers it better than a guess.
Parsing the scenario constraints
Four constraints drive the prediction:
- Top-1 routing: winner take all per token, maximum skew pressure
- No auxiliary load-balancing loss: no stabilizing force on routing distribution
- Expert 0 at 52% after 5k steps: already past collapse threshold (~40–50% triggers concern in production)
- 17× gradient ratio vs Expert 7: quantified learning-rate asymmetry
Reading the numbers. A uniform 8-expert distribution would give each expert 12.5% of tokens. Expert 0 at 52% is already 4× its fair share. Expert 1 at 31% is the only meaningful competitor, but the 21-point gap to Expert 0 is widening, not stable. Experts 2–7 each under 3% receive so few gradient updates that their weights cannot improve on the token patterns they occasionally see. The 17× gradient ratio versus Expert 7 is not a snapshot: it compounds every step. Over 10k additional steps without intervention, Expert 0 accumulates roughly 170k effective updates versus 10k for Expert 7. The router converts that gap into even higher softmax scores on common tokens. Ten thousand more steps with the same data and no intervention gives the loop ample time to compound.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Shazeer et al. (2017) report that without balancing, the gating network converges to a few favored experts.
- Production MoE training runs intervene when any expert exceeds 50% share. This scenario is past that threshold.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would adding aux loss with α=0.01 change your prediction?
Skew still widens but slower: Expert 0 might reach 60–65% instead of 75–85%. The α value determines the magnitude.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Predicting the share stays stable at 52%, which ignores the self-reinforcing gradient asymmetry.
60 second bullets to scan on the way to the call.
Positive feedback loop direction
17× gradient update asymmetry
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.