MoE replaces dense FFN with a router, N expert FFNs, and top-k weighted combination, attention stays dense.
Imagine a restaurant instead of one big kitchen cooking every dish the same way. MoE has a menu router that reads your order, picks the two best specialist chefs from a team of eight, and blends their dishes. The dining room where everyone shares news (attention) stays the same, only the kitchen (FFN) gets the specialist setup.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Naming the three components that replace a dense FFN is the minimum viable MoE vocabulary for any transformer interview. The flashcard front asks for a crisp list; the depth behind that list, how each component works, where it sits in the block, and what stays unchanged, separates memorization from understanding.
Standard transformer blocks alternate dense self-attention with dense FFN processing. MoE modifies only the FFN step, replacing a single shared MLP with a routing pipeline that conditionally activates specialist FFNs. Every production MoE LLM, Mixtral, DeepSeek, Switch, GShard, Llama 4 MoE, follows this template with variations in k, N, and balancing mechanisms.
This deep dive walks through each of the three components, shows how they connect, and clarifies what the MoE recipe does not change.
The sections below build from intuition to production practice. Read actively: after each section, pause and restate the key point in your own words, that rehearsal is what converts reading into interview-ready recall.
Component 1: The router (gating network)
The router decides which experts handle each token. It is typically a learned linear projection from the hidden state to N logits: router_logits = x @ W_router. Softmax or sigmoid converts logits to routing scores.
Softmax gating treats experts as competing choices, scores sum to 1 across experts. Sigmoid gating (used in some frontier models) allows independent per-expert activation probabilities. The router is lightweight compared to expert FFNs, its cost is O(d × N) vs O(d × f) for FFN matmuls.
During training, auxiliary load-balancing losses often regularize router behavior to prevent collapse. DeepSeek-V3 uses bias adjustment instead of aux loss. The router is the control plane; experts are the data plane.
Some models add a shared expert FFN that always runs alongside routed experts, a fourth always-on component in the FFN slot. Mention it as an extension when discussing DeepSeek or Qwen3 MoE variants.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral 8×7B: linear router, 8 expert FFNs, top-2 weighted combination per token.
- Switch Transformer: top-1 routing with capacity-factor buffers on expert dispatch.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the difference between softmax and sigmoid gating?
Softmax competes experts (sum to 1); sigmoid allows independent per-expert activation scores.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Listing only the router and experts without mentioning top-k selection and weighted combination.
60 second bullets to scan on the way to the call.
Router scores experts per token
N expert FFNs in the bank
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.