Find the wrong claim about what MoE replaces in a transformer block
Click any words you think contain an error. Click again to unmark.
MoE sparsifies the FFN sublayer only; self-attention stays dense and mixes across all token positions.
Think of a transformer block as two steps in a meeting. First, every word looks at every other word to share context (attention, everyone talks to everyone). Second, each word gets processed individually through a small neural network (the FFN). MoE replaces that second step with a team of specialist FFNs and a router, but the first step, where words share information, stays the same.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Knowing which sublayer MoE replaces is foundational transformer literacy. The spot-error claim, that MoE replaces self-attention with sparse expert routing, swaps the two sublayers entirely. It is a common slip among people who hear "sparse transformer" and assume sparsity applies everywhere.
Every standard transformer block alternates two operations: self-attention mixes information across the sequence, and the feed-forward network transforms each position independently. MoE modifies only the second operation. Understanding why attention stays dense, and why that design choice persists across Mixtral, DeepSeek, Switch, and GShard, separates surface-level MoE awareness from architecture fluency.
This deep dive maps the block recipe, explains the functional split between attention and FFN, and clarifies how MoE fits into the FFN slot without touching the attention path.
The sections below build from intuition to production practice. Read actively: after each section, pause and restate the key point in your own words, that rehearsal is what converts reading into interview-ready recall.
Anatomy of a transformer block
A decoder transformer block (GPT-style) contains, in order: layer norm → self-attention → residual add → layer norm → FFN → residual add. The attention sublayer computes weighted sums over all token representations, each position gathers context from the full sequence (causally masked in decoders).
The FFN sublayer applies the same two-layer MLP independently at each position: expand dimension, nonlinearity, project back. It is position-wise, token i's FFN output depends only on token i's hidden state after attention.
MoE replaces this FFN sublayer with a gating router plus N expert FFNs. Top-k experts run per token; outputs combine with router weights. Attention sublayers before and after MoE FFN layers remain standard dense multi-head self-attention.
Some research explores MoE attention (experts as heads or sparse attention patterns), but production LLM MoE at Mistral, DeepSeek, and Meta follows FFN-only sparsity. Mention research directions only if the interviewer asks about future architectures.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Mixtral 8×7B keeps dense sliding-window + full attention while MoE-sparsifying FFN layers.
- DeepSeek-V3 uses dense MLA attention with fine-grained MoE FFN experts.
What an interviewer would ask next. Try answering before peeking at the approach.
QCould you apply MoE to attention instead of FFN?
Possible in research (multi-head as experts) but non-standard; breaks the production recipe interviewers expect.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming MoE replaces self-attention instead of the feed-forward sublayer.
60 second bullets to scan on the way to the call.
Transformer block = attention + FFN
MoE targets FFN sublayer
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.