How does RoPE differ structurally from sinusoidal positional encoding?

Self-attention is permutation invariant: without position info, 'dog bites man' and 'man bites dog' look identical. Positional encoding fixes that. The original 2017 Transformer used **sinusoidal PE**, a fixed sin/cos vector ADDED to the input embedding. BERT used learned PE, a trainable vector per position, also added to the input. Both modify the input and leave it to the attention's projections to make sense of position. **RoPE** takes a structurally different approach. It does NOT add anything to the input. Right before the attention matmul, it ROTATES pairs of dimensions in Q and K by an angle that depends on the token's position. For position m, dimension pair (2i, 2i+1) is rotated by angle `θ_m = m / 10000^(2i/d)`. The payoff: when you compute `q_m · k_n`, the rotations combine so the result depends only on the RELATIVE position `(m - n)`. The score between tokens at positions 5 and 7 is computed the same way as between positions 105 and 107. That is what you want for language: meaning depends on distance, not absolute index. RoPE is the default in Llama 4, Mistral Large 3, Qwen 3, DeepSeek V4, Gemma 4.

**RoPE** rotates pairs of dimensions in Q and K by a position-dependent angle: $$\theta_m = m / 10000^{2i/d}$$ It does not add anything to the input embedding the way sinusoidal or learned PE does. The rotation lands right before the attention matmul, on Q and K only. V and softmax are untouched. Because rotations compose by angle subtraction, the dot product `q_m · k_n` picks up a phase that depends only on `(m - n)`. That gives `QKᵀ` a direct relative position property baked into the score, instead of one the projections have to learn to extract from an embedding plus-PE sum. Three practical wins fall out. Zero trainable parameters, so the PE doesn't grow with max context. No hard cap at training position, so extension is possible (with PI or YaRN to handle angles past the trained range). Clean kernel integration: FlashAttention 2 and 3 fuse the rotation into the attention kernel with no extra HBM round trip. This is why nearly every modern open weight decoder LLM (Llama 4, Mistral Large 3, Qwen 3, DeepSeek V4, Gemma 4) ships RoPE.