Match each positional encoding scheme to where it enters the architecture

Match pairs·Medium·4.0 · 0·~2 min·Relevant atMeta Mistral AI

Attempt it

Sinusoidal PE (Vaswani 2017)

Learned absolute PE (early BERT, GPT-2)

RoPE (rotary)

ALiBi

T5 relative bias

Five positional encoding schemes, three injection points. Match each to where in the architecture it actually enters. **Input-side schemes (enter once at the embedding).** *Sinusoidal PE* (Vaswani 2017) computes a fixed sin/cos pattern per position and adds it to the token embedding. The model never re-sees position info; it propagates only through the residual stream. *Learned absolute PE* (early BERT, GPT-2) replaces the fixed sinusoidal table with a learnable `(max_position, d_model)` lookup. Same injection point as sinusoidal: add to the token embedding at input. The only difference is whether the table is hand-designed or learned. **Per-layer Q/K-side schemes.** *RoPE* rotates the `Q` and `K` vectors inside every attention layer by a position-dependent angle. The input embedding stays position-free. RoPE re-injects fresh position information at every layer, which is what gives it strong length-extrapolation properties. **Per-layer score-side schemes.** *ALiBi* adds a per-head linear distance penalty directly to the pre-softmax attention score: `score_ij - m_h · |i - j|`. No input-side embedding involved. *T5 relative bias* adds a learned scalar bias to the pre-softmax score, indexed by which relative-distance bucket `(i - j)` falls into. Three injection points: input embedding, Q/K vectors, attention scores. Each scheme picks one.

**The mapping splits cleanly across three injection points.** **Input-side, once at the embedding.** Sinusoidal PE and learned absolute PE both add a position-specific vector to the token embedding at input. From there, the position information rides the residual stream into every block. After the input layer, attention has no fresh position information; it sees only the residual-stream representation of whatever the input layer encoded. The difference between sinusoidal and learned absolute is only whether the position-to-vector function is fixed (sinusoidal `sin(pos/10000^(2i/d))`) or learned (a `(max_position, d_model)` parameter table). Both have the same architectural injection point. **Per-layer, on Q and K.** RoPE applies a position-dependent rotation matrix to `Q` and `K` inside every attention layer, before the score matmul. The input embedding has no position component. Mathematically, after rotation `q_m · k_n` depends only on `(m - n)`, giving RoPE its relative-position property without needing a separate relative-position formalism. The critical fact: position is injected fresh at every layer. This is part of why RoPE extrapolates better than learned absolute (which is locked to the training-time `max_position`). **Per-layer, on the attention scores directly.** ALiBi and T5 relative bias both modify the pre-softmax score matrix `S = QK^T / √d_k`. ALiBi subtracts a per-head linear distance penalty: `S_ij = q_i · k_j / √d_k - m_h · |i - j|`. T5 adds a learned scalar bias indexed by which relative-distance bucket `(i - j)` falls into. Neither scheme touches the input embedding or the Q/K vectors. The position signal is purely in the score-matrix bias term. Three injection points, three architectural philosophies. Modern LLMs (Llama, Mistral, Qwen, DeepSeek, Claude) overwhelmingly use RoPE because per-layer injection extrapolates further than input-once.

Quick reference

TL;DR

Sinusoidal and learned-absolute PE enter once at the input embedding; RoPE rotates Q/K inside every attention layer; ALiBi and T5 relative bias add directly to pre-softmax attention scores in every layer.

Memory aid

Easy to grasp

Transformers don't know token order on their own; attention treats words like a bag, not a list. So you inject position information somewhere, and the choice of *where* matters as much as the choice of *what*. The old way (Vaswani 2017, BERT) was to stamp position numbers onto each word at the entry door, then never mention position again. Modern models (Llama, Mistral) rotate the query and key vectors inside every attention room by a position-dependent amount: fresher position info at every layer. ALiBi and T5 take a third route: don't touch the words at all, just add a 'how far apart are these two tokens' penalty to every attention score directly.

Key concepts

Transformer architecture Attention

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: name the three injection points, place each scheme, mention that modern LLMs converged on RoPE (per-layer Q/K).

Side-by-side

Scheme	Injection point	Per-layer or once	Length extrapolation
Sinusoidal PE	Token embedding (input)	Once	Poor in practice
Learned absolute	Token embedding (input)	Once	Hard cap at max_position
RoPE	Q and K vectors (per layer)	Every layer	Excellent with PI/YaRN
ALiBi	Pre-softmax attention score	Every layer	Excellent (built-in)
T5 relative bias	Pre-softmax attention score	Every layer	Good (logarithmic buckets)

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 8B/70B/405B and Llama 4 Maverick/Scout all use RoPE with `base=500000` (Llama 3+) and YaRN scaling for long context: per-layer Q/K injection.
BERT-base (2018) used learned absolute PE with `max_position = 512`: hard cap, no extrapolation; ModernBERT (2024) switched to RoPE for the same reason every other modern model did.
BLOOM (2022) and MPT-7B (2023) used ALiBi for length extrapolation; both showed that per-head linear distance bias works well past training length.
T5 (2020) used relative position bias as a learned scalar table per head: score-side injection but a discrete bucket lookup rather than a continuous formula.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does RoPE extrapolate better than learned absolute PE?

QCan you combine RoPE with ALiBi in the same model?

QWhat changes if you set RoPE's `base` from 10000 to 500000?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking RoPE adds to the input embedding (it doesn't; it rotates Q and K inside attention), or thinking ALiBi modifies the token vectors (it doesn't; it biases the attention scores directly).

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.