Top 50 · Transformer Architecture22 of 38

What does 'position-wise' actually mean for the transformer FFN sublayer?

MCQ·Medium·4.0 · 0·~1 min·Relevant atMistral AI

Attempt it

**Position-wise has a precise technical meaning.** The same FFN weight set is applied independently to every token in the sequence. There is no cross-token interaction inside the FFN. **What the FFN sees and does.** The input to the FFN is a tensor of shape `(batch, seq_len, d_model)`. The FFN consists of two linear layers with an activation between (the original 2017 design): ``` hidden = activation(W_up . x + b_up) out = W_down . hidden + b_down ``` The weight matrices W_up and W_down are shared across all positions. The bias vectors are shared too. The same operation is applied to position 0, position 1, ..., position seq_len-1, completely independently. If you flattened seq_len into the batch dimension, treating each token as a separate example, the FFN's output would be exactly the same. That is the test for 'position-wise': can you reshape the sequence axis into the batch axis without changing the result? **Why this matters.** All cross-token information transfer in a transformer happens in attention. The FFN does no positional reach at all; it is a per-token feature transformer. - A token at position 5 can read information from a token at position 100 only by going through the attention sublayer. - The FFN can transform a single token's vector arbitrarily complex ways, but it cannot move information between tokens. **Common confusion.** People sometimes think 'position-wise' means 'with positional encoding' or 'varying by position'. It means neither. It means the operation is the same recipe applied independently at each position.

**Position-wise means independent per-token application of the same FFN.** Three commitments follow: 1. **The weights are shared.** W_up, W_down, and the bias vectors are the same for every position. There is exactly one FFN per layer, not one FFN per position. 2. **The operation is independent.** Position i's output depends only on position i's input. No cross-token influence. 3. **No information flows between positions.** Attention is the only mechanism for that. FFN cannot move information across the sequence axis. **The test.** Take an input tensor of shape `(batch, seq_len, d_model)`. Reshape it to `(batch * seq_len, d_model)`. Run the FFN. Reshape back. You get exactly the same result. The FFN is invariant under sequence-axis collapsing because it operates per token. **Why this division of labor exists.** Attention is O(n^2) in sequence length: it must consider every pair of positions, which is expensive. The FFN is O(n) in sequence length: just n independent applications. Putting all cross-token mixing in attention and all per-token transformation in the FFN gives a clean asymptotic split. If the FFN did cross-token mixing too, you would either duplicate attention's work (wasteful) or use a different mixing pattern (which is essentially what convolutions do, and is the architectural choice CNNs made instead of transformers). The transformer's signature design is: attention for mixing, FFN for transformation, alternating in a stack. **Why the other options are wrong.** - *Option A: weights per position.* Inverts the meaning. Per-position weights would mean the FFN does NOT generalize across positions and would explode in parameter count (multiply by seq_len). The whole point of position-wise is that the same weights work everywhere. - *Option C: one output broadcast.* Wrong shape.

Quick reference

TL;DR

Position-wise means the same FFN weight set is applied independently to every token in the sequence.

Memory aid

Easy to grasp

Imagine a transformer block as a two-stage assembly line. The first stage, attention, is a group meeting where every word talks to every other word and gathers context. The second stage, the FFN, is a private booth where each word goes in alone, gets reshaped according to a fixed recipe, and comes back out. Crucially, the recipe is the same for every word; nothing about the booth changes between word 1 and word 100. Also, two words in two adjacent booths never see each other. The booth is private and per-word. The booth's only job is to transform a single word; the group meeting was where the cross-talk happened.

Key concepts

Feed-forward network Attention Position-wise operation

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: precise meaning of position-wise, the (B, T, D) reshape test, why attention and FFN have different roles, the O(n) vs O(n^2) asymptotic split, mechanistic implication for FFN-as-memory, MoE as the natural extension.

Side-by-side

Property	Attention sublayer	FFN sublayer (position-wise)
Cross-token mixing	Yes, reads and writes across positions	No, operates per position
Cost in seq_len	O(n^2 * d_model)	O(n * d_model * d_ff)
Weight sharing across positions	Yes (Q, K, V, O matrices shared)	Yes (W_up, W_down shared)
Permutation behavior	Permutation-equivariant given positional encoding	Permutation-equivariant
Per-token output depends on	All tokens in the sequence	Only that token
Role in mechanistic story	Routing / cross-token information transfer	Per-token knowledge retrieval and transformation

Real-world examples

Real products, models, and research that use this idea.

PyTorch's nn.Linear applied to a (B, T, D) tensor is automatically position-wise: it operates on the last dimension and broadcasts across batch and sequence.
FlashAttention is concerned only with the attention sublayer; the FFN sublayer needs no special kernel because its position-wise nature already saturates GPU tensor cores.
MoE designs like Mixtral and DeepSeek route each token independently to a subset of expert FFNs. This per-token routing only makes sense because the FFN is position-wise.
Mechanistic interpretability work (Geva et al. 2021 on FFN as key-value memory; Anthropic's circuits work) all relies on the position-wise FFN structure to define per-token 'features' cleanly.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf the FFN is position-wise, why is it not just called a 'token-wise MLP'?

QHow does this design relate to mechanistic interpretability claims that 'FFNs store facts' and 'attention does routing'?

QWhy do MoE designs put the experts at the FFN sublayer and not at the attention sublayer?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking 'position-wise' means the weights vary by position. It means the opposite: the weights are SHARED across positions and the operation is applied INDEPENDENTLY at each position.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.