Spot the error: 'The FFN mixes information across tokens'
Click any words you think contain an error. Click again to unmark.
The sentence reverses the two roles: attention mixes tokens, the FFN does not. The FFN is position-wise: the same MLP runs on every token's vector independently.
Think of a transformer block as two rooms in sequence. In the first room (attention), every token is allowed to look at every other token and copy whatever it needs. This is the social room, where mixing happens. In the second room (the FFN), every token sits in its own private booth and runs the same recipe on whatever it brought from the first room. No looking around, no copying. The original sentence got the rooms swapped: it claimed the booths share information and the social room is private. The actual design is the opposite, and it matters because the booth design is what makes the FFN cheap to parallelize.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: identify the swap, name the cost split, mention that MoE depends on the FFN being per-token.
Real products, models, and research that use this idea.
- Llama 3.1 8B uses a position-wise SwiGLU FFN with three matrices; the same weights run on every token in the sequence independently.
- Llama 4 Maverick's MoE block only works because the FFN is already per-token; each token can route to a different expert without breaking the rest of the block.
- FlashAttention only accelerates the attention sublayer because it is the cross-position operation; FFN runs as a plain tensor-parallel matmul, no special kernel needed.
- GPT-2's original code labels the sublayer `mlp` precisely to flag that it is a per-token MLP and not a cross-token operation.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the FFN does not mix tokens, why is it ~2/3 of the block's parameters?
QCould you build a transformer block with only attention and no FFN?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming any sublayer with the word 'network' in its name mixes tokens. The FFN is a 'position-wise feed-forward network'; the qualifier is the entire point.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.