RoPE, spell out the acronym and name exactly which vectors it rotates

Attempt it

RoPE stands for Rotary Position Embedding. It applies a position-dependent rotation to the Q and K vectors before the attention dot product. The rotation is done in pairs of adjacent feature dimensions: each pair is rotated by an angle that grows linearly with the token position and decays with frequency across feature dimensions. The V vectors are left untouched. Because both Q and K are rotated, the dot product q . k becomes a function of the relative offset between the two tokens, which is the property that gives RoPE its relative-position behavior and its decent length-extrapolation when combined with NTK scaling, position interpolation, or YaRN.

Your answer

Quick reference

TL;DR

RoPE = Rotary Position Embedding. It rotates Q and K vectors (not V) by position-dependent angles before the attention dot product, making the score depend on (m - n).

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture two compasses, one pointing in the direction of a query token and another pointing in the direction of a key token. RoPE spins each compass needle by an amount that depends on where the token sits in the sentence: token at position 5 gets spun by 5 units, token at position 10 by 10 units. When the two needles are compared (dot product), what matters is the angle BETWEEN them, which is just (10 - 5 = 5 units). So the comparison ends up depending only on how far apart the tokens are, not on their absolute positions. That is the whole RoPE trick.

Key concepts

RoPE (Rotary Position Embedding)Positional encoding Relative position

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Expand the acronym, identify Q and K as the rotated tensors (V untouched), explain the position-dependent rotation per dimension pair, show how the dot product becomes a function of (m - n), name 2026 LLMs that use it.

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick uses RoPE with high base for 128k-1M native context support.
Mistral Large 2 uses RoPE inside its sliding-window attention.
Qwen 3.5 uses RoPE with YaRN-style frequency scaling for long context.
DeepSeek V4 uses RoPE wrapped inside MLA (Multi-head Latent Attention).
The original RoFormer paper (Su et al. 2021) introduced RoPE; adoption was incremental until Llama 1 in 2023 made it mainstream.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk algebraically through why rotating Q at position m and K at position n makes the dot product a function of (m - n).

Sign in or go Pro to see the approach for this follow-up.

QWhy does Llama 3 use base=500000 instead of the RoFormer default of 10000?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying RoPE rotates the embeddings or applies to V. RoPE rotates only Q and K, after their projections, before the attention dot product.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.