Contrast absolute and relative positional encoding by what the score depends on

**Positional encoding** tells the attention layer where each token sits in the sequence. Without it, attention is permutation-equivariant, it cannot distinguish 'dog bites man' from 'man bites dog'. There are two families: absolute and relative. **Absolute** schemes attach a position-dependent vector to each token at the INPUT. Examples: sinusoidal PE (the original 2017 transformer) and learned PE (BERT). The vector for position 7 is fixed; whatever ends up at position 7 picks it up. The attention score `q_m . k_n` then depends on the specific values of m and n, not just on how far apart they are. **Relative** schemes make the attention score depend only on the offset `(m - n)`. Examples: T5 adds a learned bias bucket indexed by `(i - j)` to the attention logits. RoPE rotates Q and K by angles that depend on position, in a way that makes their dot product depend only on the difference. ALiBi adds a linear `-m * |i - j|` penalty to the logits. Why this matters: if your training data only had sequences up to 4k tokens, an absolute scheme has no idea what position 5000 should look like. A relative scheme just sees 'this token is 4000 away from that one', which is structurally similar to anything it saw during training. That is the foundation of modern long-context behavior.

The defining distinction is what `s_mn = q_m . k_n` is a function of. **Absolute PE** makes `s_mn` a function of m and n separately. Sinusoidal PE (original transformer) and learned PE (BERT, GPT-2) add a per-position vector to the input embedding; that vector flows into the Q and K projections, so the dot product carries information about the specific positions involved. **Relative PE** makes `s_mn` a function only of `(m - n)`. T5 adds a learned scalar bias indexed by `(i - j)` (bucketed) to the score. RoPE rotates Q and K by position-dependent angles; the dot product of the rotations factors through the difference. ALiBi adds `-m_h * |i - j|` to the logits per head with a learned slope `m_h`. Why relative wins for long context: it is translation-invariant. A pattern the model learned at training positions (100, 105) generalizes structurally to positions (5000, 5005) because both have offset 5. Absolute PE never saw position 5000 during training and has to extrapolate to a regime where its position vectors are undefined or out of distribution. This is why every modern open-weight LLM (Llama 4, Mistral, Qwen 3.5, DeepSeek V4) uses RoPE or a close variant, and why long-context extension techniques like NTK scaling, position interpolation, and YaRN all build on top of RoPE rather than on top of learned PE.