Complete the router computation from hidden state to expert logits
After the linear projection x @ W_router, MoE routers apply softmax or sigmoid scoring before top-k expert selection.
Think of the router like a receptionist with a quick scorecard: look at one token, run it through a tiny checklist, and get one score per specialist. Then those scores get turned into pick weights, either everyone competes for one shared pie, or each specialist gets their own yes/no gate before the top picks run.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every sparse FFN layer in a modern MoE transformer hinges on a tiny learned gate network that decides which expert FFNs run for each token. The fill-blank in this question targets the middle step of that pipeline, the scoring function that sits between raw logits and top-k selection. Candidates who stop at x @ W_router understand the linear projection but miss the nonlinear transform that turns unnormalized scores into usable routing weights.
That middle step matters because it encodes a design assumption about how experts relate to each other. Softmax forces experts to compete for a fixed probability budget, raising one expert's weight necessarily lowers others. Sigmoid treats each expert as an independent on/off gate, allowing multiple experts to score high without zero-sum competition. Production stacks overwhelmingly default to softmax, but sigmoid appears in multi-gate variants where complementary expert activation is the goal.
This deep dive walks the full three-stage router pipeline from hidden state to weighted expert output, explains the softmax versus sigmoid tradeoff with enough precision to survive follow-up questions, and connects the fill-blank answer to the regularization and serving concerns that appear in senior interviews.
Stage 1: linear projection to logits
The mechanism. Given a token hidden state x with dimension d_model, the router multiplies by a learned weight matrix W_router of shape [d_model, N], where N is the number of experts:
This single matmul produces N raw scalar scores, one logit per expert. W_router is intentionally the smallest learned layer in the MoE block: it has only d_model × N parameters, compared to the much larger expert FFN weight matrices that sit behind it. Yet it runs on every token at every MoE layer, so its latency and numerical behavior matter at serving scale.
At this stage, logits are unnormalized. They can be positive or negative, large or small, and their absolute scale varies across tokens and training steps. You could mechanically apply argmax or top-k directly on raw logits, and the routing decision would still be well-defined. Production stacks do not do this because the scoring function that follows serves two purposes: it converts logits into interpretable gating weights for weighted aggregation, and it constrains the weight scale so that expert outputs combine stably across tokens with different logit magnitudes.
Think of stage 1 as a quick poll: the router asks each expert how relevant it is to this token's hidden state, producing a vector of raw scores. Stage 2 is where those scores become routing weights.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral uses softmax gating with top-2 selection on 8 experts.
- Switch Transformer uses softmax with top-1 routing including null expert slots.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy renormalize softmax weights after top-k selection?
Top-k truncates the distribution; renormalizing keeps weighted sum scale stable across layers.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping the scoring step and going straight from raw logits to argmax without softmax or sigmoid normalization.
60 second bullets to scan on the way to the call.
Router pipeline stages in order
What the linear projection produces per expert
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.