RoPE = Rotary Position Embedding. It rotates Q and K vectors (not V) by position-dependent angles before the attention dot product, making the score depend on (m - n).
Picture two compasses, one pointing in the direction of a query token and another pointing in the direction of a key token. RoPE spins each compass needle by an amount that depends on where the token sits in the sentence: token at position 5 gets spun by 5 units, token at position 10 by 10 units. When the two needles are compared (dot product), what matters is the angle BETWEEN them, which is just (10 - 5 = 5 units). So the comparison ends up depending only on how far apart the tokens are, not on their absolute positions. That is the whole RoPE trick.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
RoPE (Rotary Position Embedding) is the positional encoding scheme that won the architectural convergence of modern LLMs. By the end of 2024, essentially every new open-weight transformer LLM ships with RoPE or a close variant. Understanding what RoPE is, exactly which tensors it acts on, and why it has the relative-position property is one of the most important architectural foundations for senior-level interview discussions on long-context modeling, attention design, and the difference between modern LLMs and their 2017-2020 predecessors.
This deep dive expands the acronym, identifies the precise location in the attention computation where RoPE operates, walks the rotation mechanics in detail, derives the relative-position property algebraically, explains the frequency spectrum that enables long-context extensions like YaRN and NTK scaling, and closes with the lineage of positional encoding schemes that led to RoPE's dominance.
By the end you should be able to write the rotation formula from memory, explain why only Q and K are rotated (and not V), and state confidently why Llama 3 raised base from 10000 to 500000.
Expanding the acronym and naming the tensors
RoPE stands for Rotary Position Embedding. Introduced by Su et al. in the RoFormer paper (2021), the technique was adopted by Llama 1 in early 2023 and has been the dominant choice for new LLMs since.
What gets rotated
The Q and K vectors per attention head. Specifically:
- Q at token position m: rotated by an angle proportional to m.
- K at token position n: rotated by an angle proportional to n.
- V at token position n: NOT rotated.
The rotation is applied AFTER the per-token linear projections W_Q and W_K, but BEFORE the attention dot product. In code:
Q = X @ W_Q # standard projection
K = X @ W_K
V = X @ W_V
Q_rot = apply_rope(Q, positions) # RoPE acts here
K_rot = apply_rope(K, positions) # RoPE acts here
# V is untouched, used as-is in the weighted sum
scores = (Q_rot @ K_rot.T) / sqrt(d_head)
attn_weights = softmax(scores)
out = attn_weights @ V
Why V is not rotated
V is the value content to be aggregated. Rotating V would not give any relative-position benefit (there is no symmetric partner to cancel the absolute-position component) and would distort the value vectors. The position-sensitive part of attention lives entirely in the Q-K comparison; V is the content payload.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 4 Maverick uses RoPE with high base for 128k-1M native context support.
- Mistral Large 2 uses RoPE inside its sliding-window attention.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk algebraically through why rotating Q at position m and K at position n makes the dot product a function of (m - n).
Represent each pair of adjacent dimensions as a complex number. RoPE rotation is multiplication by e^(j theta_m). The dot product of two rotated complex numbers is Re(z_q conj(z_k) e^(j theta (m - n))), where the exponent depends only on the offset. Sum across all pairs gives the per-head dot product as a function of the offset.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying RoPE rotates the embeddings or applies to V. RoPE rotates only Q and K, after their projections, before the attention dot product.
60 second bullets to scan on the way to the call.
Expand the RoPE acronym.
Identify exactly which tensors are rotated (Q and K, not V)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.