Predict how expert-0's routing share evolves when it already receives 52% of tokens and no load-balancing loss is active.

Attempt it

An 8-expert MoE layer trains with top-1 routing and no auxiliary load-balancing loss. After 5k steps the per-expert token fractions are: Expert 0 = 52%, Expert 1 = 31%, Experts 2–7 each under 3%. Expert 0 therefore receives roughly 17× more gradient updates per step than Expert 7. Training continues for another 10k steps with the same data distribution and no balancing regularizer. Predict Expert 0's token fraction at step 15k.

Quick reference

TL;DR

Without load-balancing pressure, Expert 0's 52% share widens past 70% over 10k more steps via the routing-collapse positive feedback loop.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a tutoring center where one teacher already handles half the students and gets better every week. The leading tutor already teaches half the class and gets better every day. With no rule forcing fair distribution, more students pick the best tutor, who improves even faster. After months, one tutor handles most of the school.

Key concepts

Mixture of Experts Fine-Tuning

Quick scan · 2–4 min

Everything important, quickly.

7 min: scenario parsing + feedback loop math + prediction calibration + wrong-answer analysis.

Real-world examples

Real products, models, and research that use this idea.

Switch Transformer ablations show routing skew worsening over training without aux loss.
Production MoE training runs intervene when any expert exceeds 50% share, this scenario is past that threshold.
DeepSeek-V3's bias adjustment exists precisely to prevent the trajectory this scenario describes.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would adding aux loss with α=0.01 change your prediction?

Sign in or go Pro to see the approach for this follow-up.

QWould top-2 instead of top-1 change the predicted trajectory?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.