Five positional encoding schemes, three injection points. Match each to where in the architecture it actually enters.
**Input-side schemes (enter once at the embedding).**
*Sinusoidal PE* (Vaswani 2017) computes a fixed sin/cos pattern per position and adds it to the token embedding. The model never re-sees position info; it propagates only through the residual stream.
*Learned absolute PE* (early BERT, GPT-2) replaces the fixed sinusoidal table with a learnable `(max_position, d_model)` lookup. Same injection point as sinusoidal: add to the token embedding at input. The only difference is whether the table is hand-designed or learned.
**Per-layer Q/K-side schemes.**
*RoPE* rotates the `Q` and `K` vectors inside every attention layer by a position-dependent angle. The input embedding stays position-free. RoPE re-injects fresh position information at every layer, which is what gives it strong length-extrapolation properties.
**Per-layer score-side schemes.**
*ALiBi* adds a per-head linear distance penalty directly to the pre-softmax attention score: `score_ij - m_h · |i - j|`. No input-side embedding involved.
*T5 relative bias* adds a learned scalar bias to the pre-softmax score, indexed by which relative-distance bucket `(i - j)` falls into.
Three injection points: input embedding, Q/K vectors, attention scores. Each scheme picks one.
**The mapping splits cleanly across three injection points.**
**Input-side, once at the embedding.** Sinusoidal PE and learned absolute PE both add a position-specific vector to the token embedding at input. From there, the position information rides the residual stream into every block. After the input layer, attention has no fresh position information; it sees only the residual-stream representation of whatever the input layer encoded.
The difference between sinusoidal and learned absolute is only whether the position-to-vector function is fixed (sinusoidal `sin(pos/10000^(2i/d))`) or learned (a `(max_position, d_model)` parameter table). Both have the same architectural injection point.
**Per-layer, on Q and K.** RoPE applies a position-dependent rotation matrix to `Q` and `K` inside every attention layer, before the score matmul. The input embedding has no position component. Mathematically, after rotation `q_m · k_n` depends only on `(m - n)`, giving RoPE its relative-position property without needing a separate relative-position formalism.
The critical fact: position is injected fresh at every layer. This is part of why RoPE extrapolates better than learned absolute (which is locked to the training-time `max_position`).
**Per-layer, on the attention scores directly.** ALiBi and T5 relative bias both modify the pre-softmax score matrix `S = QK^T / √d_k`. ALiBi subtracts a per-head linear distance penalty: `S_ij = q_i · k_j / √d_k - m_h · |i - j|`. T5 adds a learned scalar bias indexed by which relative-distance bucket `(i - j)` falls into.
Neither scheme touches the input embedding or the Q/K vectors. The position signal is purely in the score-matrix bias term.
Three injection points, three architectural philosophies. Modern LLMs (Llama, Mistral, Qwen, DeepSeek, Claude) overwhelmingly use RoPE because per-layer injection extrapolates further than input-once.
Quick reference
TL;DR
Sinusoidal and learned-absolute PE enter once at the input embedding; RoPE rotates Q/K inside every attention layer; ALiBi and T5 relative bias add directly to pre-softmax attention scores in every layer.
Transformers don't know token order on their own; attention treats words like a bag, not a list. So you inject position information somewhere, and the choice of *where* matters as much as the choice of *what*. The old way (Vaswani 2017, BERT) was to stamp position numbers onto each word at the entry door, then never mention position again. Modern models (Llama, Mistral) rotate the query and key vectors inside every attention room by a position-dependent amount: fresher position info at every layer. ALiBi and T5 take a third route: don't touch the words at all, just add a 'how far apart are these two tokens' penalty to every attention score directly.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Permutation-invariant tasks (set transformers) If your input is a set with no order, you should not add positional encoding at all. Some research architectures intentionally skip it.
Very short context tasks where learned absolute fits within training cap Learned absolute is computationally cheaper and slightly higher quality at training length; if you will never exceed `max_position`, the extra RoPE complexity is unnecessary.
Mixing schemes naively Adding learned absolute on top of RoPE has been shown to degrade quality (double-encoding of position). Pick one scheme per model.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name the three injection points, place each scheme, mention that modern LLMs converged on RoPE (per-layer Q/K).
Side-by-side
Scheme
Injection point
Per-layer or once
Length extrapolation
Sinusoidal PE
Token embedding (input)
Once
Poor in practice
Learned absolute
Token embedding (input)
Once
Hard cap at max_position
RoPE
Q and K vectors (per layer)
Every layer
Excellent with PI/YaRN
ALiBi
Pre-softmax attention score
Every layer
Excellent (built-in)
T5 relative bias
Pre-softmax attention score
Every layer
Good (logarithmic buckets)
Real-world examples
Real products, models, and research that use this idea.
1Llama 3.1 8B/70B/405B and Llama 4 Maverick/Scout all use RoPE with `base=500000` (Llama 3+) and YaRN scaling for long context: per-layer Q/K injection.
2BERT-base (2018) used learned absolute PE with `max_position = 512`: hard cap, no extrapolation; ModernBERT (2024) switched to RoPE for the same reason every other modern model did.
3BLOOM (2022) and MPT-7B (2023) used ALiBi for length extrapolation; both showed that per-head linear distance bias works well past training length.
4T5 (2020) used relative position bias as a learned scalar table per head: score-side injection but a discrete bucket lookup rather than a continuous formula.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does RoPE extrapolate better than learned absolute PE?
Sign in or go Pro to see the approach for this follow-up.
QCan you combine RoPE with ALiBi in the same model?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes if you set RoPE's `base` from 10000 to 500000?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking RoPE adds to the input embedding (it doesn't; it rotates Q and K inside attention), or thinking ALiBi modifies the token vectors (it doesn't; it biases the attention scores directly).