RoPE, spell out the acronym and name exactly which vectors it rotates

**RoPE** stands for **Rotary Position Embedding**. It is used in essentially every modern open-weight LLM (Llama 1-4, Mistral, Qwen, DeepSeek, Gemma). ### Where it acts After Q and K projections but BEFORE the attention dot product, RoPE rotates each Q and K vector. The V vectors are NOT rotated. ### What the rotation does Each pair of adjacent dimensions in a Q (or K) vector is treated as a 2D point and rotated by an angle that depends on the token position m and the dimension-pair index i: - Larger m -> larger angle. - Larger i -> slower rotation rate per position. The formula: `theta_{m, i} = m / base^{2i/d}`, with `base = 10000` originally (Llama 3 raised it to 500000). ### Why both Q and K get rotated Because both vectors rotate symmetrically, the dot product `q . k` becomes a function only of the OFFSET `m - n`, not of the absolute positions. This is the relative-position property: shifting the whole sequence does not change the attention pattern. A pattern learned at training positions (100, 105) generalizes to evaluation positions (5000, 5005) because both have offset 5.

**RoPE = Rotary Position Embedding** (Su 2021). Dominant in modern transformer LLMs. ### Where it acts After Q and K projections, before the attention dot product. V skips RoPE entirely. ### What the rotation looks like Each Q (or K) vector of dimension d_head is split into d_head/2 pairs of adjacent dimensions. Each pair is rotated as a 2D point by angle: $$\theta_m = m / \text{base}^{2i/d}$$ where m is position, i is the dimension pair, base is the hyperparameter (10000 originally, 500000 in Llama 3+ for long context). ### Why both Q and K get rotated Rotating Q at position m by theta_m and K at position n by theta_n makes the dot product factor through `(m - n)`. This gives RoPE the relative-position property: shifting the sequence does not change attention scores. ### Why V is not rotated V is the value to aggregate, not a participant in the position-sensitive comparison. Rotating V would not contribute to the relative-position property (no symmetric partner to cancel absolute position) and would distort values. ### Why it dominates 2026 LLMs Relative-position property enables long-context extension via NTK scaling, PI, and YaRN. No learned position parameters. Multiplicative (rotation) composes naturally with attention's matmul. No hard cap on sequence length.