Compare absolute and relative positional encoding. What does the score depend on in each case?

**Positional encoding** (PE) tells the attention layer where each token sits in the sequence. Without it, attention is permutation-equivariant. It cannot distinguish 'dog bites man' from 'man bites dog'. **Absolute** schemes attach a position-dependent vector to each token at the INPUT. Examples: sinusoidal PE (the original 2017 transformer) and learned PE (BERT). The vector for position 7 is fixed; whatever ends up at position 7 picks it up. The attention score `q_m . k_n` then depends on the specific values of m and n, not just on how far apart they are. **Relative** schemes make the attention score depend only on the offset `(m - n)`. Examples: T5 adds a learned bias bucket indexed by `(i - j)` to the attention logits. RoPE rotates Q and K by angles that depend on position, in a way that makes their dot product depend only on the difference. Suppose your training data only had sequences up to 4k tokens. An absolute scheme has no idea what position 5000 should look like. A relative scheme sees 'this token is 4000 away from that one', which is structurally similar to anything it saw during training. That is the foundation of modern long-context behavior.

The defining distinction is what `s_mn = q_m . k_n` is a function of. **Absolute PE** makes `s_mn` a function of m and n separately. Sinusoidal PE (original transformer) and learned PE (BERT, GPT-2) add a per position vector to the input embedding. That vector flows into the Q and K projections, so the dot product carries information about the specific positions involved. **Relative PE** makes `s_mn` a function only of `(m - n)`. T5 adds a learned scalar bias indexed by `(i - j)` (bucketed) to the score. RoPE rotates Q and K by position-dependent angles; the dot product of the rotations factors through the difference. ALiBi adds `-m_h * |i - j|` to the logits per head with a fixed slope `m_h`. Why relative wins for long context: it is translation-invariant. A pattern the model learned at training positions (100, 105) generalizes structurally to positions (5000, 5005) because both have offset 5. Absolute PE never saw position 5000 during training and has to extrapolate to a regime where its position vectors are undefined or out of distribution. This is why most modern open-weight LLMs (Llama 3, Mistral, Qwen3, DeepSeek-V3) use RoPE or a close variant. It is also why long-context extension techniques like NTK scaling, position interpolation, and YaRN all build on top of RoPE rather than on top of learned PE.