A model vendor's marketing claims their MoE has a dedicated 'math expert.' Your team is choosing models for a quantitative product and wants the claim tested. Design an audit that would confirm or refute semantic specialization, and state what published routing analyses would lead you to expect.
Audit with three probes: routing statistics against the balanced k/N baseline, causal ablation of the named expert, and surface feature controls. Published analyses predict token level patterns, not a clean math expert.
Picture a mail sorting office that claims sorter number 3 handles all the legal mail. To test that, you would do three things. Watch the conveyor for a week and count which letters actually land on each desk, comparing against what pure chance would give. Then send sorter 3 on vacation and see whether legal mail specifically piles up while everything else flows fine. Finally, check whether sorter 3 is really reading the letters or just grabbing every envelope with a fancy wax seal, by mailing legal letters in plain envelopes and junk mail in fancy ones. If the plain legal letters scatter across other desks, the office sorts by envelope, not by content. Model audits work the same way: count, remove, and control for the superficial cue.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A dedicated math expert is one of the most intuitive stories you can tell about a Mixture of Experts model, which is exactly why marketing teams tell it. The architecture seems to invite it: separate FFNs, a router choosing between them, surely the model organizes itself by subject the way a company organizes departments.
The interview question is really asking whether you can convert a fuzzy mechanistic claim into falsifiable measurements. That requires three distinct kinds of evidence: distributional evidence that routing actually concentrates, causal evidence that the concentration matters, and adversarial evidence that the signal is not a cheap surface cue wearing a math costume.
It also requires knowing the literature well enough to state a prior. Routing analyses have been published for Mixtral, ST-MoE, OLMoE, and others, and they tell a consistent story that is far less tidy than the marketing version. Walking through the audit design, you should be able to predict the likely outcome before running a single token.
Set the null hypothesis: what balanced routing looks like
Every audit needs a baseline, and MoE routing has a natural one. Training applies a load-balancing pressure, an auxiliary loss in the Switch lineage or bias adjustments in DeepSeek-V3's loss free scheme, that pushes each expert toward an equal share of traffic. With N experts and top-k routing, the expected share of routed tokens per expert is:
For Mixtral's 8 experts with top-2, that is 25% of tokens touching any given expert. This matters because naive observers see the math expert receives 30% of math tokens and gasp, when the balanced expectation was already 25%. Enrichment must be measured as distance from k/N, not from zero.
The null hypothesis for the audit: on math input, the claimed expert's share is statistically indistinguishable from its share on matched control input. The alternative the vendor is claiming: a large, layer consistent excess on math input specifically.
Two refinements make the comparison honest. Weight by routing probability rather than raw counts if the gate is soft, since a barely second place pick differs from a dominant one. And compute the statistics per layer from the start. A 32 layer model makes 32 independent routing decisions per token, and what the router keys on changes with depth, so a single pooled number can manufacture or hide the effect.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral's published routing analysis across The Pile found no expert owning a domain; the strongest patterns were syntactic and positional, with consecutive tokens frequently routed to the same experts.
- OLMoE from AI2 released router logs and analysis tooling openly, showing early layer routing keyed heavily on token identity rather than topic.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you quantify routing concentration rigorously rather than eyeballing shares?
Compare observed per expert distributions to the balanced one with KL divergence or a chi square test, bootstrapping over documents for confidence intervals.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating high routing share on math corpora as proof of semantic specialization, without controlling for surface features like digits and symbols and without causal ablation.
60 second bullets to scan on the way to the call.
What balanced baseline share should routing statistics be compared against?
Why does routing concentration alone fail to prove semantic specialization?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.