A model provider's marketing claims their MoE has a dedicated 'math expert.' Your team is choosing models for a quantitative product and wants the claim tested. Design an audit that would confirm or refute semantic specialization, and state what published routing analyses would lead you to expect.
Audit with three probes: routing statistics against the balanced k/N baseline, causal ablation of the named expert, and surface feature controls. Published analyses predict token level patterns, not a clean math expert.
Picture a mail sorting office that claims sorter number 3 handles all the legal mail. To test that, you would do three things. Watch the conveyor for a week and count which letters land on each desk, comparing against what pure chance would give. Then send sorter 3 on vacation and see whether legal mail specifically piles up while everything else flows fine. Finally, check whether sorter 3 reads the letters or just grabs every envelope with a fancy wax seal. To do that, mail legal letters in plain envelopes and junk mail in fancy ones. If the plain legal letters scatter across other desks, the office sorts by envelope, not by content. Model audits work the same way: count, remove, and control for the superficial cue.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A dedicated math expert is one of the most intuitive stories you can tell about a Mixture of Experts model. That is why marketing teams tell it. The architecture seems to invite it. There are separate FFNs and a router choosing between them, so surely the model organizes itself by subject, the way a company organizes departments.
The task is to convert a fuzzy mechanistic claim into falsifiable measurements. That needs three kinds of evidence. Distributional evidence shows that routing concentrates. Causal evidence shows that the concentration matters. Adversarial evidence shows that the signal is not a cheap surface cue that only looks like math.
It also needs knowledge of the literature, enough to state a prior. Researchers have published routing analyses for Mixtral, ST-MoE, OLMoE, and others. Their findings are far less tidy than the marketing version. Walking through the audit design, you should be able to predict the likely outcome before running a single token.
Set the null hypothesis: what balanced routing looks like
Every audit needs a baseline, and MoE routing has a natural one. Training applies a load-balancing pressure that pushes each expert toward an equal share of traffic. It is an auxiliary loss in the Switch lineage, or bias adjustments in DeepSeek-V3's loss free scheme. With N experts and top-k routing, the expected share of routed tokens per expert is:
For Mixtral's 8 experts with top-2, that is 25% of tokens touching any given expert. This matters because naive observers see that the math expert receives 30% of math tokens and are impressed, when the balanced expectation was already 25%. Enrichment must be measured as distance from k/N, not from zero.
The null hypothesis for the audit: on math input, the claimed expert's share is statistically indistinguishable from its share on matched control input. The alternative the vendor is claiming: a large, layer consistent excess on math input specifically.
Two refinements make the comparison honest. Weight by routing probability rather than raw counts if the gate is soft, since a barely second place pick differs from a dominant one. And compute the statistics per layer from the start. A 32 layer model makes 32 independent routing decisions per token, and what the router keys on changes with depth. So a single pooled number can create or hide the effect.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral's published routing analysis across The Pile found no expert owning a domain. The strongest patterns were syntactic and positional, with consecutive tokens frequently routed to the same experts.
- OLMoE from AI2 released router logs and analysis tooling openly. It showed clear domain specialization and token-identity specialization that grows in later layers, while Mixtral stayed close to uniform.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you quantify routing concentration rigorously rather than eyeballing shares?
Compare observed per expert distributions to the balanced one with KL divergence or a chi square test, bootstrapping over documents for confidence intervals.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating high routing share on math corpora as proof of semantic specialization, without controlling for surface features like digits and symbols and without causal ablation.
60 second bullets to scan on the way to the call.
What balanced baseline share should routing statistics be compared against?
Why does routing concentration alone fail to prove semantic specialization?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.