Contrast ALiBi's linear distance bias with RoPE's vector rotation, which one modifies scores additively, and which one modifies Q and K directly?
ALiBi: additive per-head bias on pre-softmax scores. RoPE: rotation applied to Q and K vectors before the dot product. Different stages, both relative.
Picture a classroom where students at the back hear the teacher less clearly. ALiBi is like installing the same kind of volume drop in everyone's ears, the speaker stays the same, but distant listeners hear softer audio by a fixed amount per row. RoPE is different. Imagine the teacher and each student wear a turntable on their head, and the turntable spins by an amount that depends on which seat they are in. The lecturer's voice arrives clearly only when the two turntables are spinning in compatible ways, which depends on the difference between their seats. ALiBi adjusts how loud each link sounds. RoPE adjusts how each speaker and each listener are oriented before they connect.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
ALiBi and RoPE are both 2021-vintage relative positional schemes that survive in production today, but they intervene at different points of the attention computation. Mistaking one for the other on an interview question is a tell that the candidate has only memorized labels.
This deep dive lays out the two schemes side by side: where each one hooks in, the math of each operation, why both reach a relative-position property despite using opposite mechanisms, and how the choice cascades into production tradeoffs like length extrapolation and kernel composition.
Mental model: ALiBi is one tensor add after QK^T. RoPE is a rotation applied to Q and K before they ever meet. Same destination, completely different route.
Pipeline stages: where each scheme intervenes
The standard attention pipeline
A single attention head walks through these steps, in order:
- Project the input through W_Q, W_K, W_V to get Q, K, V.
- Compute the score matrix:
scores = QK^T / sqrt(d_k). - Optionally add an additive bias (masking, position bias).
- Apply softmax row-wise to get attention weights.
- Compute the context vector:
out = weights @ V.
Where ALiBi hooks in
ALiBi acts at step 3. It adds a precomputable matrix B_ij = -m_h * |i - j| to the score matrix. Embeddings, Q, K, V are completely untouched. The only line of code that changes is the one right before softmax.
Where RoPE hooks in
RoPE acts between step 1 and step 2. After Q and K are computed by their projections, RoPE rotates each pair of dimensions of Q at position m by an angle theta_m, and the corresponding pair in K at position n by theta_n. The QK^T matmul that follows is now between rotated vectors, and the relative-position property emerges from the rotation algebra.
Why the staging matters
ALiBi being post-matmul means it composes with any attention kernel that supports an additive bias, no kernel rewrite required. RoPE being pre-matmul means the kernel must apply the rotation before the score computation; FlashAttention 2 and 3 do this natively but it is a different integration story.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | ALiBi | RoPE |
|---|---|---|
| Pipeline stage | Score matrix (after QK^T) | Q and K vectors (before dot product) |
| Operation | Additive bias | Rotation (multiplicative) |
| Parameters | Zero (fixed slopes) | Zero (fixed base frequency) |
| Length extrapolation | Native, with receptive-field cap | Needs PI / NTK / YaRN extension |
| Modern adoption | BLOOM, MPT, some efficient stacks | Llama, Mistral, Qwen, DeepSeek, Gemma |
Real products, models, and research that use this idea.
- BLOOM 176B (BigScience, 2022) and MPT-7B/30B (MosaicML, 2023) ship with ALiBi.
- Llama 1 through Llama 4 Maverick, Mistral, Qwen 3.5, DeepSeek V4, and Gemma 4 use RoPE.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does RoPE produce a relative property without explicitly subtracting positions?
Walk through the rotation algebra. A rotation by angle theta_m on q and theta_n on k means the dot product after rotation is the original dot product evaluated at a rotation by (theta_m - theta_n), which is a function of (m - n) only.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming both schemes act at the same stage of the pipeline. They do not. ALiBi acts on the scalar score matrix; RoPE transforms Q and K before the dot product even runs.
60 second bullets to scan on the way to the call.
Where in the attention pipeline ALiBi acts
Where in the attention pipeline RoPE acts
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.