Attention is permutation-equivariant, it sees a set, not a sequence. Positional encoding injects the order attention can't see on its own.
Imagine someone hands you a bag of Scrabble tiles spelling out D, O, G, and asks if the word is DOG or GOD. Without knowing the order, you can't say. Attention sees its inputs the same way: a bag of tiles, no first or last. Positional encoding is like writing a tiny number on each tile so the reader can put them back in order before answering.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The permutation-equivariance of attention is one of the most important things to internalize about transformers, it explains why positional encoding is non-optional and provides the conceptual foundation for understanding RoPE, ALiBi, and the long context extrapolation literature.
This deep dive treats the question through a mathematical lens: state the formal property, derive why it makes attention a set operation, walk through the four PE schemes as different strategies for breaking the symmetry, and finish with the connection to Deep Sets and Set Transformer.
The payoff is a clean mental model, attention is a set function, PE turns it into a sequence function, that explains every downstream choice in the long context literature.
The formal property and its derivation
For any permutation matrix P:
The derivation is mechanical. The score matrix s_ij = q_i · k_j / √d_k transforms as a permuted matrix under P. Softmax is row-wise so it commutes with row permutation. The value weighted sum is a sum over an index, and sums are order independent.
What this rules out
The operation cannot, by construction, produce any output that depends on absolute position. There's no place inside the formula where an index could enter: every aggregation is symmetric.
Equivariance is a structural property of the formula, not a quirk of any particular implementation. You cannot 'fix' it by tweaking the kernel; the symmetry has to be broken upstream.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| PE scheme | Mechanism | Extrapolation | Used in |
|---|---|---|---|
| Sinusoidal | Fixed sin/cos added to embedding | Limited | Original Transformer |
| Learned absolute | Trained vector per position | None past trained length | BERT, GPT-2 |
| RoPE | Rotate Q, K by position-dependent angle | Good with base tuning | Llama 4, Mistral, DeepSeek V4, Qwen 3.5 |
| ALiBi | Linear bias to scores by |i−j| | Excellent | MPT, BLOOM |
Real products, models, and research that use this idea.
- Llama 4 Maverick and DeepSeek V4 use RoPE with a tuned base frequency for long context extrapolation past 1M tokens.
- Mistral and Gemma 4 use RoPE paired with sliding-window attention for bounded per-layer cost.
What an interviewer would ask next. Try answering before peeking at the approach.
QSome recent papers show that decoder-only LMs can train without positional encoding (NoPE): how is that possible?
Causal masking itself breaks the permutation symmetry: token at position i only attends to positions ≤ i, so positions 0 and 5 see structurally different contexts. The model can in principle learn to count via this mask asymmetry. In practice NoPE works on some tasks but is less robust than RoPE or ALiBi, especially for length extrapolation, and doesn't apply to encoder-only models at all.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing token embedding (which handles vocabulary identity) with positional encoding (which handles position identity). They're different problems addressed by different mechanisms.
60 second bullets to scan on the way to the call.
The formal permutation-equivariance statement and what it implies
Difference between equivariance and invariance
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.