An 8-expert MoE layer trains with top-1 routing and no auxiliary load balancing loss. After 5k steps the per expert token fractions are: Expert 0 = 52%, Expert 1 = 31%, Experts 2–7 each under 3%. Expert 0 therefore receives roughly 17× more gradient updates per step than Expert 7. Training continues for another 10k steps with the same data distribution and no balancing regularizer. Predict Expert 0's token fraction at step 15k.
Without load-balancing pressure, Expert 0's 52% share widens past 70% over 10k more steps via the routing-collapse positive feedback loop.
Imagine a tutoring center where one teacher already handles half the students and gets better every week. The leading tutor already teaches half the class and gets better every day. With no rule forcing fair distribution, more students pick the best tutor, who improves even faster. After months, one tutor handles most of the school.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This predict-output scenario is a calibrated judgment test on MoE training dynamics. You are given concrete numbers, 52% leader share, 17× gradient ratio, top-1 routing, no aux loss, 10k more steps, and must predict the direction and magnitude of change. The question punishes candidates who treat routing share as static and rewards those who understand the positive feedback loop.
The expected answer (>70%, typically 75–85%) is not a precise measurement but a range consistent with collapse dynamics observed in Switch Transformer ablations and production monitoring dashboards.
This deep dive parses the scenario, walks the feedback math, calibrates the prediction, and explains why common wrong answers fail.
The scenario deliberately removes every stabilizing mechanism, no aux loss, no capacity factor, no bias adjustment, so you must reason from first principles about whether the loop accelerates or stalls. Production teams see this pattern in real training runs: a dashboard shows one expert crossing 50% share by step 5k, and the question is whether to intervene now or wait. This deep dive gives you the quantitative tools to answer that judgment call with a defensible range, not a guess.
Parsing the scenario constraints
Four constraints drive the prediction:
- Top-1 routing, winner take all per token, maximum skew pressure
- No auxiliary load-balancing loss, no stabilizing force on routing distribution
- Expert 0 at 52% after 5k steps, already past collapse threshold (~40–50% triggers concern in production)
- 17× gradient ratio vs Expert 7, quantified learning-rate asymmetry
Experts 2–7 each under 3% are effectively dying, they receive so few updates they cannot improve on common patterns. Expert 1 at 31% is the only meaningful competitor, but the gap to Expert 0 is widening. Ten thousand more steps with the same data and no intervention gives the loop ample time to compound.
Reading the numbers. A uniform 8-expert distribution would give each expert 12.5% of tokens. Expert 0 at 52% is already 4× its fair share. Expert 1 at 31% is the only meaningful competitor, but the 21-point gap to Expert 0 is widening, not stable. Experts 2–7 each under 3% are in a death spiral: they receive so few gradient updates that their weights cannot improve on the token patterns they occasionally see. The 17× gradient ratio versus Expert 7 is not a snapshot, it compounds every step. Over 10k additional steps without intervention, Expert 0 accumulates roughly 170k effective updates versus 10k for Expert 7, a gap that the router converts into even higher softmax scores on common tokens.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Switch Transformer ablations show routing skew worsening over training without aux loss.
- Production MoE training runs intervene when any expert exceeds 50% share, this scenario is past that threshold.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would adding aux loss with α=0.01 change your prediction?
Skew still widens but slower, Expert 0 might reach 60–65% instead of 75–85%; α tuning determines magnitude.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Predicting the share stays stable at 52%, that ignores the self-reinforcing gradient asymmetry.
60 second bullets to scan on the way to the call.
Positive feedback loop direction
17× gradient update asymmetry
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.