Mixture-of-Experts is a surgical change to a transformer block. It replaces only the **FFN sublayer**, and leaves attention, residual connections, and normalization untouched.
In a dense block, the FFN is a single MLP: two or three matrices that every token passes through. In an MoE block, that single MLP is replaced by `N` parallel expert FFNs (8, 64, 128; the number varies by model). A small linear layer called the **router** scores each expert for each token, and the top-k experts (usually k=1 or k=2) are actually run on that token.
Because only `k` of `N` experts run per token, the FLOPs stay close to a dense FFN. But because all `N` expert weight sets exist, total parameters grow roughly `N`x. The router's softmaxed scores combine the chosen experts' outputs into one vector that replaces what the dense FFN would have produced.
The block recipe is otherwise identical: same pre-norm, same residual add. MoE is a sublayer swap, not a block rewrite.
**MoE is a sublayer-local swap.** The attention sublayer, residual wrap, and normalization placement are all unchanged. Only the dense FFN gets replaced.
The replacement has two pieces. First, `N` parallel expert FFNs, each with the same shape as the dense FFN it replaces. Second, a tiny router (a single `(d_model, N)` linear layer plus softmax) that scores experts per token and selects the top-k.
**Sparse activation is the entire point.** With `k=1` (Llama 4 Maverick, 128 experts, top-1) or `k=2` (Mixtral, 8 experts, top-2), each token runs through only a small fraction of the total parameters. The active FLOP count stays close to a dense FFN of the same per-expert size, while the total parameter count scales with `N`.
**Two recurring misreadings.** First, MoE does not touch attention or the block scaffolding; the residual stream, pre-norm, and attention layer are bit-identical to a dense block. Second, sparse routing means tokens see only `k` experts, not all of them. If every token went through every expert (dense routing), there would be no FLOP savings and no reason to bother.
The router learns which expert each token should visit through a load-balancing auxiliary loss during training, so expert utilization stays roughly uniform.
Quick reference
TL;DR
MoE swaps the dense FFN for N expert FFNs plus a tiny router; attention and the block scaffolding are untouched.
Imagine a transformer block as a factory with two stations. The first station (attention) lets every item see every other item. The second station used to be one big general-purpose machine (the dense FFN); MoE replaces it with eight smaller specialist machines plus a clerk who decides which machine each item should visit. Most items only go to one or two of the eight, so the factory runs about as fast as before; but the total floor space (parameters) is eight times bigger, which means the factory now stores eight times as much know-how. Llama 4 Maverick takes this to 128 specialists, with each item visiting just one.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Edge or single-GPU deployment with tight HBM budget Even though MoE saves active FLOPs, all expert weights must sit in HBM. A 128-expert MoE needs roughly 128x the FFN weight memory of dense, fatal on a single consumer GPU.
Tiny model regime (under 1B parameters) At small scale, the router and load-balancing overhead exceed the quality gains; dense FFN is more parameter-efficient below ~1B.
Latency-critical serving with batch size 1 Sparse routing fragments the matmul into per-expert sub-matmuls, hurting GPU utilization at batch 1; dense FFN keeps the matmul clean.
Quick scan · 2–4 min
Everything important, quickly.
4 min: name the swap (FFN only), describe the router, give the param/FLOP arithmetic with a real model (Mixtral or Maverick), mention load-balancing loss.
Side-by-side
Block component
Dense block
MoE block
Attention sublayer
Standard MHA/GQA/MLA
Unchanged (same MHA/GQA/MLA)
Pre-norm placement
Before each sublayer
Unchanged
Residual wrap
x + sublayer(norm(x))
Unchanged
FFN sublayer body
One MLP, all tokens
N expert MLPs + router, k per token
FFN params per block
~135M (Llama-7B sized)
~135M × N (e.g., 128× for Maverick)
Active FLOPs per token (FFN)
1× dense FFN
k × dense FFN (k=1 or 2)
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick uses 128 experts with top-1 routing in every FFN sublayer; attention is standard GQA, block scaffolding is pre-norm + residual just like Llama 3.1.
2Mixtral 8x7B (Mistral AI) uses 8 experts per layer with top-2 routing; the '8x' in the name refers to the FFN expansion, not to 8 separate models.
4Qwen2.5-MoE uses an FFN-local MoE swap with load-balancing loss; the attention sublayer uses standard GQA identical to dense Qwen models.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy doesn't MoE replace attention as well as the FFN?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens to the auxiliary load-balancing loss after training?
Sign in or go Pro to see the approach for this follow-up.
QIf MoE saves FLOPs, why is it not the universal architecture?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming MoE touches attention or replaces the block recipe. The swap is local to the FFN sublayer; everything else (residual wrap, pre-norm, attention) is identical to a dense block.