Why has RoPE become the dominant positional encoding in modern LLMs?

Self-attention has no built in notion of order. The original Transformer used **sinusoidal PE** (a fixed sin/cos vector added to the input embedding). BERT used **learned PE** (a trainable vector per position, also added to the input). Both work on the input side and leave it to W_q and W_k to figure out position-aware behavior. **RoPE** takes a different route. It doesn't touch the input. Right before the attention dot product, it rotates pairs of dimensions in Q and K by a position-dependent angle: $$\theta_m = m / 10000^{2i/d}$$ The payoff is direct relative position behavior. When you dot `q_m` with `k_n`, the two rotations combine and the score ends up depending only on `(m - n)`. 'cat sat' next to each other behaves the same way whether the pair sits at positions 5,6 or 105,106. Three more wins: zero trainable parameters, no hard max position cap, and a cheap rotation that fuses into FlashAttention kernels. Add the empirical edge and you get total adoption: Llama 4, Mistral Large 3, Qwen 3, DeepSeek V4, Gemma 4, every modern open weight decoder.

Direct relative position attention is what **RoPE** buys you. It rotates Q and K by position-dependent angles, and because rotations compose by angle addition, the dot product becomes a function of relative distance: $$q_m \cdot k_n = f(m - n)$$ Three alternatives sit alongside this: - Sinusoidal PE gives the same property only indirectly, W_q and W_k have to learn to extract it from the embedding plus-PE sum. - Learned PE has a hard cap at max training position and doesn't generalize past it. - RoPE has no learned parameters, no hard cap, and slots cleanly into FlashAttention. Extension past training length is handled by PI or YaRN. Adoption is universal in modern open weight decoders: Llama 1/2/3/4, Mistral, Mixtral, Qwen, DeepSeek, GPT-J.