ALiBi = Attention with Linear Biases. It adds a per head linear distance penalty to pre softmax scores; no input position embedding.
Imagine a group chat where every person hears every other person, but each listener wears a different pair of headphones. Some headphones get louder the closer the speaker sits; others barely change with distance. The chat itself has no name tags, no seating chart, no announcement of who is two seats away from whom. The volume curves are the only source of position information. ALiBi is that volume curve applied to attention: instead of telling tokens where they sit, it just turns down the volume on far away tokens by a fixed amount that grows with distance. Different attention heads use different volume curves, so some heads focus on nearby tokens and others still hear the whole room.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Positional information in transformers can be injected at three plausible places in the pipeline. ALiBi picks the third, the score matrix, and the choice is the load bearing thing about the scheme.
This deep dive walks through what ALiBi stands for, where the bias lives in the attention computation, why the slope schedule is per head and geometric, what gives the technique its length-extrapolation property, and how it compares to the schemes that have largely replaced it at frontier scale.
Mental model: ALiBi adds one line of code. After
scores = QK^T / sqrt(d_k), you add a precomputable per head bias matrixB_ij = -m_h * |i - j|, then softmax as usual.
Expanding the acronym and locating the injection point
What the letters mean
ALiBi expands to Attention with Linear Biases. The two key words to internalize are Linear (the bias is linear in distance) and Bias (it is an additive term, not a rotation, not an embedding).
The three candidate injection points
A transformer has three obvious places to inject position information:
- Input embeddings: add a position vector to each token before any attention math.
- Q and K vectors: rotate or otherwise modify the projected vectors before the dot product.
- Score matrix: add a per position term to
QK^T / sqrt(d_k)before softmax.
Sinusoidal PE and learned PE pick option one. RoPE picks option two. ALiBi picks option three.
Why the score matrix choice matters
Injecting at the score matrix means the model touches embeddings and Q and K exactly the same way it would without ALiBi. The only change is one tensor add right before softmax. This is what makes ALiBi simple to retrofit into any attention implementation and what makes the bias trivially compatible with FlashAttention-style fused kernels.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- BLOOM 176B (BigScience, 2022) shipped with ALiBi as its positional scheme.
- MPT-7B and MPT-30B (MosaicML, 2023) used ALiBi and advertised length extrapolation past training context.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did frontier labs move from ALiBi to RoPE plus extension tricks?
RoPE preserves the relative property too but does so multiplicatively in the Q/K space, which empirically scales to larger models without the receptive field cap. Extension schemes like PI and YaRN gave the field a knob to push context length without retraining.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying ALiBi modifies the input embeddings or the Q/K vectors. It does neither. The bias lives at the score matrix, one line after the QK^T multiplication.
60 second bullets to scan on the way to the call.
What does the acronym ALiBi expand to
Where in the attention pipeline does ALiBi inject position
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.