What does the router inside a Mixture-of-Experts block actually do, per token?
The router is a tiny linear layer projecting from the model width to N expert scores per token, picking the top-k (usually 1 or 2) and weighting their FFN outputs by softmaxed scores.
Imagine a busy hospital where every patient walks in and a single receptionist instantly decides which one or two specialists they need to see, out of a hundred specialists in the building. The receptionist is the router: tiny compared to a specialist (one quick form, not a full exam), but their decision controls all the heavy work. If the receptionist always sent everyone to the same two specialists, the other ninety-eight would sit idle, so the hospital trains the receptionist to spread the load evenly. In an MoE transformer, the router does exactly this for every token in every block.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The router is the smallest component in an MoE block and the one that makes the design economical. Every other piece (the expert FFNs, the load-balancing loss, the sparse-activation FLOP savings) exists to support what the router decides. Without a router, MoE is just N parallel dense FFNs running on every token, which would be Nx more expensive than dense, not cheaper.
This question asks for the mechanics: shape, decision, and effect on what runs downstream. A good answer is concrete about the linear-layer dimensions and the top-k selection, not vague about 'gating'.
Router shape and the per-token decision
The router is a single linear layer projecting from the model width to N expert scores, where N is the number of experts in the block. There is typically no bias, no activation, no normalization inside the routing path. For each token vector, the router computes:
A top-k selection extracts the k highest-scoring experts; their logits are softmaxed (the other N minus k scores are dropped). The result is a sparse weight vector with k non-zero entries summing to 1.
Only the k selected expert FFNs are run on the token. Their outputs are weighted by the softmaxed scores and summed:
This is the entire FFN sublayer output for that token. Position-wise, per-token, with the only nonlocal decision being which k of N experts get the work.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 4 Maverick: router scores 128 experts, picks top-1, runs only that expert per token (`k/N = 1/128` sparsity).
- Mixtral 8x7B: router scores 8 experts, picks top-2, runs both with softmaxed weights (`k/N = 2/8 = 0.25` sparsity).
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat happens if you increase k from 1 to 2 to 4 at fixed N?
Active FLOPs grow linearly with k; quality typically improves up to k=2 and plateaus. Higher k erodes the sparsity savings without proportional quality gains.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the router as expensive or stateful. It is a single tiny linear layer with no recurrence and no awareness of other tokens; the whole point is that routing decisions are cheap and local.
60 second bullets to scan on the way to the call.
Describe the router's shape (a small linear from model width to expert count) and what it emits per token
Top-k selection values used in production models (k=1 for Maverick, k=2 for Mixtral)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.