Leading collapse indicators: high per-expert token CV (>0.5), falling router entropy, and one expert dominating >40% while others idle.
Routing collapse is like one checkout lane getting all the customers while others stand empty: you notice the long queue before the store's daily revenue drops. CV measures how uneven the lanes are, entropy tracks whether the automatic lane-picker is funneling everyone to the same place, and a single lane above 40% is an obvious warning sign.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Which observability metrics flag MoE routing collapse before validation loss moves is an operations question that tests whether you understand collapse as a load-distribution failure, not a task-quality failure, at least initially. The multi-select format pairs three leading indicators against three plausible but wrong distractors.
Routing collapse is one of MoE's most dangerous failure modes because it wastes model capacity silently. A few experts do all the work while others atrophy. Validation loss may stay flat for thousands of steps because the dominant expert compensates, until effective capacity shrinks enough that quality drops suddenly.
This deep dive explains the collapse loop, validates each correct metric, and explains why the distractors are lagging or irrelevant.
The sections below build from intuition to production practice. Read actively: after each section, pause and restate the key point in your own words, that rehearsal is what converts reading into interview-ready recall.
The routing collapse positive-feedback loop
Collapse begins when one expert randomly receives slightly more tokens early in training:
- Expert i gets more tokens → more gradient updates.
- Expert i improves faster → higher quality outputs for routed tokens.
- Router learns to prefer expert i → higher routing scores.
- Expert i receives even more tokens → loop accelerates.
- Other experts receive near-zero tokens → stop learning → atrophy.
Effective model capacity shrinks from N experts toward 1. This is a load-balancing failure detectable in dispatch telemetry long before language modeling loss reflects the reduced capacity.
Build dashboards with per-expert histograms updated every N steps, humans spot dominant-expert patterns faster than automated thresholds alone. Automate alerts on CV and fraction thresholds for overnight runs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Google Switch/GShard training pipelines alert on per-expert token fraction skew before loss anomalies.
- DeepSeek MoE training dashboards track router entropy and CV as primary routing health metrics.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat CV threshold triggers intervention vs healthy specialization?
CV 0.3–0.5 often healthy; >0.5 investigate; >0.8 likely collapse. Context matters.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using validation loss or GPU memory as early collapse detectors instead of utilization skew metrics.
60 second bullets to scan on the way to the call.
CV of per-expert token counts
Router softmax entropy trend
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.