Attention is permutation-equivariant: it sees a set, not a sequence. Positional encoding injects the order attention can't see on its own.
Imagine someone hands you a bag of Scrabble tiles spelling out D, O, G, and asks if the word is DOG or GOD. Without knowing the order, you can't say. Attention sees its inputs the same way: a bag of tiles, no first or last. Positional encoding is like writing a tiny number on each tile so the reader can put them back in order before answering.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The permutation-equivariance of attention explains why positional encoding is required. It is also the conceptual foundation for understanding RoPE, ALiBi, and the long-context extrapolation literature.
Attention is a set function, and PE turns it into a sequence function. That mental model explains every downstream choice in the long-context literature.
The formal property and its derivation
For any permutation matrix P:
The derivation is mechanical. The score matrix s_ij = q_i · k_j / √d_k transforms as a permuted matrix under P. Softmax is row-wise so it commutes with row permutation. The value weighted sum is a sum over an index, and sums are order independent.
What this rules out
The operation cannot, by construction, produce any output that depends on absolute position. There's no place inside the formula where an index could enter: every aggregation is symmetric.
Equivariance is a structural property of the formula, not a quirk of any particular implementation. You cannot 'fix' it by tweaking the kernel; the symmetry has to be broken upstream.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| PE scheme | Mechanism | Extrapolation | Used in |
|---|---|---|---|
| Sinusoidal | Fixed sin/cos added to embedding | Limited | Original Transformer |
| Learned absolute | Trained vector per position | None past trained length | BERT, GPT-2 |
| RoPE | Rotate Q, K by position-dependent angle | Good with base tuning | Llama 4, Mistral, DeepSeek-V3, Qwen 3.5 |
| ALiBi | Linear bias to scores by |i−j| | Excellent | MPT, BLOOM |
Real products, models, and research that use this idea.
- Llama 4 Maverick uses RoPE in its chunked local attention layers and no positional encoding in its global layers.
- Mistral 7B v0.1 and Gemma 3 use RoPE paired with sliding-window attention for bounded per-layer cost.
What an interviewer would ask next. Try answering before peeking at the approach.
QSome recent papers show that decoder-only LMs can train without positional encoding (NoPE): how is that possible?
Causal masking itself breaks the permutation symmetry: token at position i only attends to positions ≤ i, so positions 0 and 5 see structurally different contexts. The model can in principle learn to count via this mask asymmetry. In small-scale studies NoPE has done well (Kazemnejad et al. 2023 found it length-generalized better than explicit schemes on their tasks). But most large models keep at least some explicit PE, and NoPE doesn't apply to encoder-only models at all.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing token embedding (which handles vocabulary identity) with positional encoding (which handles position identity). They're different problems addressed by different mechanisms.
60 second bullets to scan on the way to the call.
The formal permutation-equivariance statement and what it implies
Difference between equivariance and invariance
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.