RoPE = Rotary Position Embedding. It rotates Q and K vectors (not V) by position-dependent angles before the attention dot product, making the score depend on (m - n).
Picture two compasses, one pointing in the direction of a query token and another pointing in the direction of a key token. RoPE spins each compass needle by an amount that depends on where the token sits in the sentence. The token at position 5 gets spun by 5 units, and the token at position 10 by 10 units. When the two needles are compared (dot product), what matters is the angle BETWEEN them, which is just (10 - 5 = 5 units). So the comparison ends up depending only on how far apart the tokens are, not on their absolute positions. That is the whole RoPE trick.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
RoPE (Rotary Position Embedding) is the positional encoding scheme that won the architectural convergence of modern LLMs. By the end of 2024, almost every new open-weight transformer LLM ships with RoPE or a close variant.
Expanding the acronym and naming the tensors
RoPE stands for Rotary Position Embedding. Su et al. introduced it in the RoFormer paper (2021). Llama 1 adopted it in early 2023, and it has been the dominant choice for new LLMs since.
What gets rotated
RoPE rotates the Q and K vectors per attention head:
- Q at token position m: rotated by an angle proportional to m.
- K at token position n: rotated by an angle proportional to n.
- V at token position n: NOT rotated.
The rotation is applied AFTER the per token linear projections W_Q and W_K, but BEFORE the attention dot product. In code:
Q = X @ W_Q # standard projection
K = X @ W_K
V = X @ W_V
Q_rot = apply_rope(Q, positions) # RoPE acts here
K_rot = apply_rope(K, positions) # RoPE acts here
# V is untouched, used as-is in the weighted sum
scores = (Q_rot @ K_rot.T) / sqrt(d_head)
attn_weights = softmax(scores)
out = attn_weights @ V
Why V is not rotated
V is the value content to be aggregated. Rotating V would not give any relative-position benefit (there is no symmetric partner to cancel the absolute-position component) and would distort the value vectors. The position-sensitive part of attention lives entirely in the Q-K comparison; V is the content payload.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 4 Maverick uses RoPE with high base for 128k-1M native context support.
- Mistral Large 2 uses RoPE with full attention across its 128k context.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk algebraically through why rotating Q at position m and K at position n makes the dot product a function of (m - n).
Represent each pair of adjacent dimensions as a complex number. RoPE rotation is multiplication by e^(j theta_m). The dot product of two rotated complex numbers is Re(z_q conj(z_k) e^(j theta (m - n))), where the exponent depends only on the offset. Sum across all pairs gives the per-head dot product as a function of the offset.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying RoPE rotates the embeddings or applies to V. RoPE rotates only Q and K, after their projections, before the attention dot product.
60 second bullets to scan on the way to the call.
Expand the RoPE acronym.
Identify exactly which tensors are rotated (Q and K, not V)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.