The router is a tiny learned linear layer that, for each token, scores every expert and picks the top-k (typically k=1 or k=2). Only those k expert FFNs are run on that token's vector; their outputs are combined with the router's softmaxed scores. Tokens at different positions can route to different experts, which is what gives MoE its sparse-FLOP property: total parameters scale with expert count, but per-token compute stays close to a single dense FFN.
The **router** is the smallest component in an MoE block and the one that makes the whole design work.
Mechanically, it is a single linear layer that projects from the model width to N expert scores, where N is the number of experts. For each token's vector, it produces an N-dimensional score, applies softmax, and selects the top-k experts (usually k=1 or k=2). Only those k experts run on that token; the others are skipped entirely.
A concrete trace. Mixtral has 8 experts and uses top-2 routing. Token 5 enters the FFN sublayer as a 4096-dim vector. The router produces 8 scores like [0.05, 0.02, 0.4, 0.01, 0.03, 0.45, 0.02, 0.02]. After softmax over the top 2 (experts 2 and 5), the weights become roughly [0.47, 0.53]. Expert 2 runs on the token producing one output, expert 5 produces another, and the sublayer output is the softmax-weighted sum of the two.
Different tokens route to different experts. Token 6 might pick experts 0 and 7. That per-token choice is what gives MoE its sparse-FLOP property.
**The router does two jobs per token: score and select.**
It is a single `(d_model, N)` linear layer: no bias, no normalization, no nonlinearity inside the routing decision. For each token, it produces an `N`-dimensional logit vector. The top-k entries (usually `k=1` for Llama 4 Maverick with 128 experts, or `k=2` for Mixtral with 8) are kept; the rest are zeroed out. Softmax is applied over the selected `k` to get combination weights, and only those `k` expert FFNs are run on the token's vector.
The router's parameter cost is negligible: for Mixtral, `(4096, 8) = 32K` params per layer compared to ~1B per layer for the experts. The compute cost is also negligible: one tiny matmul per token versus the full expert FFN compute.
**The two structural facts that follow.** First, the router is *per-token*, not per-batch or per-sequence: tokens at positions 5 and 6 can route to completely different experts. Second, the routing is *learned*: during training, a load-balancing auxiliary loss pushes the router toward even expert utilization, otherwise it collapses onto a few favorites. At inference, the loss is gone but the learned routing weights remain.
The payoff is the FLOP-vs-param split: total params scale with `N`, active FLOPs scale with `k`.
Quick reference
TL;DR
The router is a tiny linear layer projecting from the model width to N expert scores per token, picking the top-k (usually 1 or 2) and weighting their FFN outputs by softmaxed scores.
Imagine a busy hospital where every patient walks in and a single receptionist instantly decides which one or two specialists they need to see, out of a hundred specialists in the building. The receptionist is the router: tiny compared to a specialist (one quick form, not a full exam), but their decision controls all the heavy work. If the receptionist always sent everyone to the same two specialists, the other ninety-eight would sit idle, so the hospital trains the receptionist to spread the load evenly. In an MoE transformer, the router does exactly this for every token in every block.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-expert blocks or fully dense FFN designs If `N = 1` there is nothing to route; the router becomes an identity and adds overhead with no benefit.
Hard token to expert assignment by rules (e.g., language or modality) Rule-based routing skips the learned router entirely. Some multilingual research uses this, but it forfeits the routing flexibility that makes MoE work for general LLMs.
Very tight latency budgets at batch 1 Even with a cheap router, sparse routing fragments matmul shapes and hurts GPU utilization at batch 1. Dense FFN is faster per request when batch size is tiny.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name the linear-layer shape, walk one token through top-k selection and softmax weighting, mention load-balancing loss.
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick: router scores 128 experts, picks top-1, runs only that expert per token (`k/N = 1/128` sparsity).
2Mixtral 8x7B: router scores 8 experts, picks top-2, runs both with softmaxed weights (`k/N = 2/8 = 0.25` sparsity).
3DeepSeek V3: 256 fine-grained experts plus 1 shared expert; routes top-8 of the 256 specialists and always runs the shared one.
4Switch Transformer (Fedus 2021): introduced top-1 routing at scale and proved load-balancing loss was essential; this is the lineage every modern MoE inherits from.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat happens if you increase k from 1 to 2 to 4 at fixed N?
Sign in or go Pro to see the approach for this follow-up.
QCould the router itself be replaced by attention over expert embeddings?
Sign in or go Pro to see the approach for this follow-up.
QHow do shared experts (DeepSeek V3) change the router's job?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the router as expensive or stateful. It is a single tiny linear layer with no recurrence and no awareness of other tokens; the whole point is that routing decisions are cheap and local.