Spot the error: 'The FFN mixes information across tokens'
Click any words you think contain an error. Click again to unmark.
The sentence reverses the two roles: attention mixes tokens, the FFN does not. The FFN is position-wise: the same MLP runs on every token's vector independently.
Think of a transformer block as two rooms in sequence. In the first room (attention), every token is allowed to look at every other token and copy whatever it needs. This is the social room, where mixing happens. In the second room (the FFN), every token sits in its own private booth and runs the same recipe on whatever it brought from the first room. No looking around, no copying. The original sentence got the rooms swapped: it claimed the booths share information and the social room is private. The actual design is the opposite, and it matters because the booth design is what makes the FFN cheap to parallelize.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Inside a standard transformer block, there are exactly two sublayers that do real work: attention and the feed-forward network. The block's design assigns one specific job to each, and the two jobs are intentionally non-overlapping. Confusing which sublayer does which is one of the most common architectural misreadings, and the question's flawed sentence is a textbook example.
The error matters more than it looks. Once you know which sublayer is the cross-token one, you also know where parameters live, where compute scales quadratically with sequence length, and why a swap like MoE only touches one of the two sublayers. Get this wrong and the entire serving-cost story comes out backward.
What the attention sublayer actually does
Attention is the cross-position operation in the block. Given the residual stream tensor of shape (sequence, model-width), the attention sublayer projects each token to Q, K, and V, computes the score matrix using scaled QK^T, applies softmax to get attention weights, and uses those weights to take a weighted sum of all V vectors in the sequence.
The critical structural fact is that the score matrix has the sequence axis appearing twice (sequence by sequence). This is what lets information move across positions in a single layer: token 5's output is a function of V vectors at every position the query is allowed to see, not just position 5's own V.
The cost grows quadratically in sequence length and linearly in model width. This is the quadratic cost transformers pay for the privilege of one-hop information movement. Every length-extrapolation trick (FlashAttention, ring attention, sparse attention, sliding window) is an attempt to reduce or amortize this quadratic factor.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3.1 8B uses a position-wise SwiGLU FFN with three matrices; the same weights run on every token in the sequence independently.
- Llama 4 Maverick's MoE block only works because the FFN is already per-token; each token can route to a different expert without breaking the rest of the block.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the FFN does not mix tokens, why is it ~2/3 of the block's parameters?
Parameter count and information-flow are different. The FFN is wide (ffn_hidden ≈ 4x d_model) because it needs capacity to do per-token nonlinear transformation; capacity does not imply mixing.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming any sublayer with the word 'network' in its name mixes tokens. The FFN is a 'position-wise feed-forward network'; the qualifier is the entire point.
60 second bullets to scan on the way to the call.
Which sublayer mixes information across positions
What 'position-wise' means in the FFN context
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.