**Position-wise has a precise technical meaning.** The same FFN weight set is applied independently to every token in the sequence. There is no cross-token interaction inside the FFN.
**What the FFN sees and does.** The input to the FFN is a tensor of shape `(batch, seq_len, d_model)`. The FFN consists of two linear layers with an activation between (the original 2017 design):
```
hidden = activation(W_up . x + b_up)
out = W_down . hidden + b_down
```
The weight matrices W_up and W_down are shared across all positions. The bias vectors are shared too. The same operation is applied to position 0, position 1, ..., position seq_len-1, completely independently.
If you flattened seq_len into the batch dimension, treating each token as a separate example, the FFN's output would be exactly the same. That is the test for 'position-wise': can you reshape the sequence axis into the batch axis without changing the result?
**Why this matters.** All cross-token information transfer in a transformer happens in attention. The FFN does no positional reach at all; it is a per-token feature transformer.
- A token at position 5 can read information from a token at position 100 only by going through the attention sublayer.
- The FFN can transform a single token's vector arbitrarily complex ways, but it cannot move information between tokens.
**Common confusion.** People sometimes think 'position-wise' means 'with positional encoding' or 'varying by position'. It means neither. It means the operation is the same recipe applied independently at each position.
**Position-wise means independent per-token application of the same FFN.** Three commitments follow:
1. **The weights are shared.** W_up, W_down, and the bias vectors are the same for every position. There is exactly one FFN per layer, not one FFN per position.
2. **The operation is independent.** Position i's output depends only on position i's input. No cross-token influence.
3. **No information flows between positions.** Attention is the only mechanism for that. FFN cannot move information across the sequence axis.
**The test.** Take an input tensor of shape `(batch, seq_len, d_model)`. Reshape it to `(batch * seq_len, d_model)`. Run the FFN. Reshape back. You get exactly the same result. The FFN is invariant under sequence-axis collapsing because it operates per token.
**Why this division of labor exists.** Attention is O(n^2) in sequence length: it must consider every pair of positions, which is expensive. The FFN is O(n) in sequence length: just n independent applications. Putting all cross-token mixing in attention and all per-token transformation in the FFN gives a clean asymptotic split.
If the FFN did cross-token mixing too, you would either duplicate attention's work (wasteful) or use a different mixing pattern (which is essentially what convolutions do, and is the architectural choice CNNs made instead of transformers). The transformer's signature design is: attention for mixing, FFN for transformation, alternating in a stack.
**Why the other options are wrong.**
- *Option A: weights per position.* Inverts the meaning. Per-position weights would mean the FFN does NOT generalize across positions and would explode in parameter count (multiply by seq_len). The whole point of position-wise is that the same weights work everywhere.
- *Option C: one output broadcast.* Wrong shape.
Quick reference
TL;DR
Position-wise means the same FFN weight set is applied independently to every token in the sequence.
Imagine a transformer block as a two-stage assembly line. The first stage, attention, is a group meeting where every word talks to every other word and gathers context. The second stage, the FFN, is a private booth where each word goes in alone, gets reshaped according to a fixed recipe, and comes back out. Crucially, the recipe is the same for every word; nothing about the booth changes between word 1 and word 100. Also, two words in two adjacent booths never see each other. The booth is private and per-word. The booth's only job is to transform a single word; the group meeting was where the cross-talk happened.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Defining 'positional encoding' for an interviewer Positional encoding is added to embeddings before any block runs; it has nothing to do with position-wise FFN.
Explaining how the FFN handles long-range dependencies The FFN does not handle long-range dependencies. Attention does. The FFN's only job is per-token feature transformation.
Estimating attention's parameter count Position-wise is a property of the FFN sublayer; attention has different sharing and dimensionality concerns.
Designing a CNN or local-attention model Those architectures explicitly use limited-range cross-position mixing in their non-attention layers; that is the opposite of position-wise.
Quick scan · 2–4 min
Everything important, quickly.
4 min: precise meaning of position-wise, the (B, T, D) reshape test, why attention and FFN have different roles, the O(n) vs O(n^2) asymptotic split, mechanistic implication for FFN-as-memory, MoE as the natural extension.
Side-by-side
Property
Attention sublayer
FFN sublayer (position-wise)
Cross-token mixing
Yes, reads and writes across positions
No, operates per position
Cost in seq_len
O(n^2 * d_model)
O(n * d_model * d_ff)
Weight sharing across positions
Yes (Q, K, V, O matrices shared)
Yes (W_up, W_down shared)
Permutation behavior
Permutation-equivariant given positional encoding
Permutation-equivariant
Per-token output depends on
All tokens in the sequence
Only that token
Role in mechanistic story
Routing / cross-token information transfer
Per-token knowledge retrieval and transformation
Real-world examples
Real products, models, and research that use this idea.
1PyTorch's nn.Linear applied to a (B, T, D) tensor is automatically position-wise: it operates on the last dimension and broadcasts across batch and sequence.
2FlashAttention is concerned only with the attention sublayer; the FFN sublayer needs no special kernel because its position-wise nature already saturates GPU tensor cores.
3MoE designs like Mixtral and DeepSeek route each token independently to a subset of expert FFNs. This per-token routing only makes sense because the FFN is position-wise.
4Mechanistic interpretability work (Geva et al. 2021 on FFN as key-value memory; Anthropic's circuits work) all relies on the position-wise FFN structure to define per-token 'features' cleanly.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the FFN is position-wise, why is it not just called a 'token-wise MLP'?
Sign in or go Pro to see the approach for this follow-up.
QHow does this design relate to mechanistic interpretability claims that 'FFNs store facts' and 'attention does routing'?
Sign in or go Pro to see the approach for this follow-up.
QWhy do MoE designs put the experts at the FFN sublayer and not at the attention sublayer?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking 'position-wise' means the weights vary by position. It means the opposite: the weights are SHARED across positions and the operation is applied INDEPENDENTLY at each position.