Which observability metrics flag MoE routing collapse before validation loss moves? · Zenaique

Attempt it

Quick reference

TL;DR

Leading collapse indicators: high per-expert token CV (>0.5), falling router entropy, and one expert dominating >40% while others idle.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Routing collapse is like one checkout lane getting all the customers while others stand empty: you notice the long queue before the store's daily revenue drops. CV measures how uneven the lanes are, entropy tracks whether the automatic lane-picker is funneling everyone to the same place, and a single lane above 40% is an obvious warning sign.

Key concepts

Mixture of Experts Pretraining Fine-Tuning

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

6 min: collapse loop + leading vs lagging metrics + CV/entropy/fraction thresholds + distractor analysis.

Real-world examples

Real products, models, and research that use this idea.

Google Switch/GShard training pipelines alert on per-expert token fraction skew before loss anomalies.
DeepSeek MoE training dashboards track router entropy and CV as primary routing health metrics.
Open-source MoE trainers (Megablocks) expose per-expert dispatch histograms for collapse detection.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat CV threshold triggers intervention vs healthy specialization?

Sign in or go Pro to see the approach for this follow-up.

QHow does collapse during fine-tuning differ from pretraining?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Using validation loss or GPU memory as early collapse detectors instead of utilization skew metrics.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.