Which observability metrics flag MoE routing collapse before validation loss moves?
Leading collapse indicators: high per-expert token CV (>0.5), falling router entropy, and one expert dominating >40% while others idle.
Routing collapse is like one checkout lane getting all the customers while others stand empty: you notice the long queue before the store's daily revenue drops. CV measures how uneven the lanes are, entropy tracks whether the automatic lane-picker is funneling everyone to the same place, and a single lane above 40% is an obvious warning sign.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
6 min: collapse loop + leading vs lagging metrics + CV/entropy/fraction thresholds + distractor analysis.
Real products, models, and research that use this idea.
- Google Switch/GShard training pipelines alert on per-expert token fraction skew before loss anomalies.
- DeepSeek MoE training dashboards track router entropy and CV as primary routing health metrics.
- Open-source MoE trainers (Megablocks) expose per-expert dispatch histograms for collapse detection.
What an interviewer would ask next. Try answering before peeking at the approach.
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Using validation loss or GPU memory as early collapse detectors instead of utilization skew metrics.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.