Contrast absolute and relative positional encoding by what the score depends on

Attempt it

**Positional encoding** tells the attention layer where each token sits in the sequence. Without it, attention is permutation-equivariant, it cannot distinguish 'dog bites man' from 'man bites dog'. There are two families: absolute and relative. **Absolute** schemes attach a position-dependent vector to each token at the INPUT. Examples: sinusoidal PE (the original 2017 transformer) and learned PE (BERT). The vector for position 7 is fixed; whatever ends up at position 7 picks it up. The attention score `q_m . k_n` then depends on the specific values of m and n, not just on how far apart they are. **Relative** schemes make the attention score depend only on the offset `(m - n)`. Examples: T5 adds a learned bias bucket indexed by `(i - j)` to the attention logits. RoPE rotates Q and K by angles that depend on position, in a way that makes their dot product depend only on the difference. ALiBi adds a linear `-m * |i - j|` penalty to the logits. Why this matters: if your training data only had sequences up to 4k tokens, an absolute scheme has no idea what position 5000 should look like. A relative scheme just sees 'this token is 4000 away from that one', which is structurally similar to anything it saw during training. That is the foundation of modern long-context behavior.

The defining distinction is what `s_mn = q_m . k_n` is a function of. **Absolute PE** makes `s_mn` a function of m and n separately. Sinusoidal PE (original transformer) and learned PE (BERT, GPT-2) add a per-position vector to the input embedding; that vector flows into the Q and K projections, so the dot product carries information about the specific positions involved. **Relative PE** makes `s_mn` a function only of `(m - n)`. T5 adds a learned scalar bias indexed by `(i - j)` (bucketed) to the score. RoPE rotates Q and K by position-dependent angles; the dot product of the rotations factors through the difference. ALiBi adds `-m_h * |i - j|` to the logits per head with a learned slope `m_h`. Why relative wins for long context: it is translation-invariant. A pattern the model learned at training positions (100, 105) generalizes structurally to positions (5000, 5005) because both have offset 5. Absolute PE never saw position 5000 during training and has to extrapolate to a regime where its position vectors are undefined or out of distribution. This is why every modern open-weight LLM (Llama 4, Mistral, Qwen 3.5, DeepSeek V4) uses RoPE or a close variant, and why long-context extension techniques like NTK scaling, position interpolation, and YaRN all build on top of RoPE rather than on top of learned PE.

Quick reference

TL;DR

Absolute PE makes the attention score depend on positions m and n separately; relative PE makes it depend only on the offset (m - n). The latter gives translation invariance and better length extrapolation.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture two people standing on a number line. Absolute thinking says 'one is at 7 and the other is at 12, the score depends on those numbers'. Relative thinking says 'they are 5 apart, that is all that matters'. If you slide both of them three steps to the right (now at 10 and 15), absolute thinking sees new numbers and recomputes; relative thinking shrugs because the gap is still 5. Modern long-context models all want the relative behavior because real language patterns care about how far apart tokens are, not where they sit in the absolute timeline.

Key concepts

Positional encoding RoPE ALiBi

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Define the absolute vs relative split by what the score is a function of, map each PE family (sinusoidal, learned, T5, RoPE, ALiBi), explain translation-equivariance and length extrapolation, name 2026 production examples.

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick, Mistral, Qwen 3.5, and DeepSeek V4 all use RoPE because of its relative-position property and long-context extensibility.
Original 2017 transformer used sinusoidal absolute PE; BERT and GPT-2 used learned absolute PE.
T5 introduced the learned relative-position bias bucket scheme that paired with its denoising pretraining.
MPT-30B and BLOOM-176B used ALiBi for its length-extrapolation property; modern frontier LLMs have moved past it for RoPE+YaRN.
Long-context extensions (YaRN at 128k, LongRoPE) all assume the RoPE relative-position structure as their starting point.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QRoPE rotates Q and K. Show algebraically that the resulting dot product depends only on (m - n).

Sign in or go Pro to see the approach for this follow-up.

QIf relative PE has all these advantages, why did the original transformer use absolute sinusoidal PE?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing where each scheme is APPLIED (input vs logits) with what each scheme is a FUNCTION OF (absolute positions vs offset). Those are independent axes.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.