Track routed-expert traffic distribution (fraction, CV, entropy) and operational failures (dropped tokens), not static parameter/FLOP ratios.
Healthy MoE monitoring is like watching checkout lanes at a store: you need to see which lanes are overloaded, how uneven traffic is, and whether customers are being turned away, not just the total number of registers installed.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MoE production failures rarely announce themselves as NaN losses. They arrive as skewed expert traffic, peaked routers, and silently dropped tokens, then later as quality regressions users report. Platform teams need metrics that move before benchmarks fail.
This multi_select separates runtime distribution signals from static model descriptors. Option E is the trap for candidates who conflate "active FLOPs" vocabulary with observability engineering.
This deep dive builds a minimal viable MoE dashboard and explains why each selected metric earns a chart.
MoE production failures arrive as skewed expert traffic, peaked routers, and silently dropped tokens, not as NaN losses. Platform teams need metrics that move before benchmarks fail. This multi_select separates runtime distribution signals (A, B, C, D) from static model descriptors (E trap). This deep dive builds the minimal viable MoE dashboard.
SRE and ML platform teams share this dashboard during MoE launches. Knowing which metrics page first, and which metric is a trap, signals you have operated MoE at scale, not just read architecture diagrams.
The E trap is deliberately chosen because active-params-over-FLOPs vocabulary appears everywhere in MoE marketing and interview prep. Candidates who confuse marketing arithmetic with ops metrics select E reflexively.
Per-expert token fraction (A)
Primary histogram. For each MoE layer, track fraction of tokens routed to expert i over a sliding window. Healthy models show uneven but not extreme specialization, collapse shows a few bars near 0.4–0.6 and many near zero.
Compare across layers: early layers often differ from late layers. Compare pre/post fine-tune: router drift shifts fractions without changing total params.
This is the first chart operators open during a MoE incident.
Histogram as primary signal. Per-expert token fraction is the first chart operators open during a MoE incident. Healthy models show uneven but not extreme specialization, collapse shows a few bars near 0.4–0.6 and many near zero. Compare across layers (early versus late), across training steps (drift over time), and pre/post fine-tune (router drift shifts fractions without changing total params).
Slice histograms by data domain when possible: code-heavy traffic may legitimately skew toward code-specialized experts. Collapse diagnosis requires distinguishing healthy specialization from pathological dominance, context matters.
Compare token fraction histograms across replicas in the same deployment. Divergence between replicas signals routing instability or uneven traffic steering, not just model behavior.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Large MoE training runs log per-expert token counts and aux-loss terms every N steps.
- Inference platforms alert on routing entropy collapse after model updates or quantization changes.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat CV threshold would you treat as a paging alert?
No universal number, baseline per model, alert on delta from healthy steady state; cite 40-60% busiest vs idlest skew from concept file.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Selecting total params divided by active FLOPs, that is a static model descriptor, not a live routing health signal.
60 second bullets to scan on the way to the call.
Per-expert token fraction histograms
CV as scalar skew summary
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.