Contrast ALiBi's linear distance bias with RoPE's vector rotation, which one modifies scores additively, and which one modifies Q and K directly?

**ALiBi** and **RoPE** both give a transformer a way to know which token is where, but they intervene at completely different points. ALiBi acts on the score matrix. After the model computes `scores = QK^T / sqrt(d_k)`, ALiBi adds a per-head bias `-m_h * |i - j|` to each entry. The bias depends only on the distance between query position i and key position j. The slope m_h is fixed per head, not learned. Q and K themselves are never touched. RoPE acts on Q and K. Before the dot product runs, each pair of dimensions in Q and K is rotated by an angle that scales with position. When you then take the dot product of a rotated query at position m with a rotated key at position n, the result depends on the difference (m - n) because rotations subtract cleanly under dot products. So the easy mental model: ALiBi adds a number to the score; RoPE spins the vectors before the score is computed. Both give relative-position behavior, but through opposite mechanisms.

**ALiBi** is additive and lives at the score matrix. **RoPE** is multiplicative and lives one step earlier, on the Q and K vectors themselves. The ALiBi modification is one tensor add. After `scores = QK^T / sqrt(d_k)`, ALiBi adds a per-head bias `B_ij = -m_h * |i - j|` and lets softmax run. The slopes m_h come from a fixed geometric series across heads, with no learned parameters. Embeddings, Q, and K are untouched. RoPE rotates Q and K before the dot product. Each pair of dimensions is rotated by a position-dependent angle: $$\theta_m = m / 10000^{2i/d}$$ When you dot a rotated `q_m` with a rotated `k_n`, the rotation algebra gives you a value that depends on `(m - n)`, not on m and n separately. That is the relative-position property, achieved at the level of the vectors instead of the level of the score. The practical consequence is that they compose differently. ALiBi sits cleanly on top of any FlashAttention kernel because it is a closed-form additive bias. RoPE sits inside the QKV pipeline and needs the kernel to apply rotation before the matmul, which most modern implementations do natively. Both yield relative position. The implementation surface they expose is different.