Pick the statements that accurately describe how an MoE sublayer replaces a dense FFN
MoE swaps the dense FFN for N expert FFNs plus a tiny router; attention and the block scaffolding are untouched.
Imagine a transformer block as a factory with two stations. The first station (attention) lets every item see every other item. The second station used to be one big general-purpose machine (the dense FFN); MoE replaces it with eight smaller specialist machines plus a clerk who decides which machine each item should visit. Most items only go to one or two of the eight, so the factory runs about as fast as before; but the total floor space (parameters) is eight times bigger, which means the factory now stores eight times as much know-how. Llama 4 Maverick takes this to 128 specialists, with each item visiting just one.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Mixture-of-Experts is the most important block-level architectural change since RMSNorm and SwiGLU. By 2026, all three open-weight frontier families (Mixtral, Llama 4, DeepSeek V3, Qwen2.5-MoE) ship MoE variants alongside their dense counterparts. Understanding what MoE actually changes inside the block (and what it leaves alone) separates a candidate who has read the architecture diagram from one who has just heard the buzzword.
The answer is narrower than people often think. MoE is a surgical sublayer swap. Attention is untouched. Pre-norm is untouched. The residual stream geometry is untouched. The change is local to the FFN sublayer body, and the rest of the block scaffolding is bit-identical to a dense block.
What MoE replaces, exactly
Take a standard pre-norm transformer block: residual stream, RMSNorm, attention, residual add, RMSNorm, FFN, residual add. In a dense block, the FFN is a single MLP, typically W_up, activation, W_down (or SwiGLU's three matrices: W_gate, W_up, W_down).
MoE replaces only the FFN body. The new sublayer holds N independent expert FFNs, each with the same shape as the dense FFN it replaced, plus a tiny router. Everything else in the block (the first RMSNorm, the attention sublayer, the residual adds, the second RMSNorm) stays bit-identical to dense. If you diff a Mixtral block against a Mistral 7B block, the only changes are inside the FFN sublayer body.
This surgical scope is why MoE was easy to adopt. Mistral did not invent a new block recipe; they took the dense block, swapped in MoE for the FFN, and shipped. Llama 4, DeepSeek V3, and Qwen2.5-MoE followed the same pattern.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Block component | Dense block | MoE block |
|---|---|---|
| Attention sublayer | Standard MHA/GQA/MLA | Unchanged (same MHA/GQA/MLA) |
| Pre-norm placement | Before each sublayer | Unchanged |
| Residual wrap | x + sublayer(norm(x)) | Unchanged |
| FFN sublayer body | One MLP, all tokens | N expert MLPs + router, k per token |
| FFN params per block | ~135M (Llama-7B sized) | ~135M × N (e.g., 128× for Maverick) |
| Active FLOPs per token (FFN) | 1× dense FFN | k × dense FFN (k=1 or 2) |
Real products, models, and research that use this idea.
- Llama 4 Maverick uses 128 experts with top-1 routing in every FFN sublayer; attention is standard GQA, block scaffolding is pre-norm + residual just like Llama 3.1.
- Mixtral 8x7B (Mistral AI) uses 8 experts per layer with top-2 routing; the '8x' in the name refers to the FFN expansion, not to 8 separate models.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy doesn't MoE replace attention as well as the FFN?
Attention is fundamentally cross-position, so 'routing' would require deciding which expert handles which (query, key) pair (combinatorial). Some research explores MoE attention (Switch Heads, MoA), but it is not the standard production design.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming MoE touches attention or replaces the block recipe. The swap is local to the FFN sublayer; everything else (residual wrap, pre-norm, attention) is identical to a dense block.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.