ALiBi injects position by penalizing far apart token pairs before softmax, with each head set to a different decay rate.
Imagine a room of listeners who all turn the volume down on speakers sitting far away. The further a speaker sits, the quieter they sound. Each listener in the room has different ears: some can only hear nearby voices, while others still catch words from across the room. The model never gets a 'seat number' for any word. Instead, the volume drop quietly tells the model who is close and who is far. And because listeners have such different hearing, some end up paying attention to neighbors only, while others can listen across the whole room. The room handles short and long range together, just from how loud each speaker sounds.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
ALiBi is the cleanest example of position as architectural bias. The model has no input side positional embedding at all. Position information enters purely through a small structural change to the attention computation, applied as a bias term right before softmax.
Understanding ALiBi pays off in two ways. First, when it shipped, ALiBi was the strongest length extrapolation story in the field. A 1k-context model held coherent perplexity at 2k, 4k, even 16k inference length, something no other PE scheme could claim without retraining. Second, the bias not embedding pattern shows up in newer variants like T5's relative attention bias and certain MoE attention designs. Knowing the original makes those easier to read.
This section walks the mechanism end to end, the per-head slope schedule, why pre-softmax placement is critical, the extrapolation argument and its limits, and finally why the open weight frontier picked RoPE+YaRN over ALiBi despite ALiBi's earlier head start.
The mechanism: one extra term in the score matrix
Start with standard scaled dot product attention. For query position i and key position j, the pre-softmax score is (q_i · k_j) / √d. ALiBi inserts one extra term before softmax:
For causal models, the absolute value reduces to (i - j) for j ≤ i, the only positions a causal mask permits. The full attention becomes softmax((QKᵀ)/√d + B).
What's not there matters as much as what is. No PE vector gets added to the input embedding. No rotation gets applied to Q or K. The token embeddings flow through the model unchanged. The only place position information lives is in that bias term inside the score matrix.
Walk through a concrete case. A 7B causal transformer with 32 heads, 4k training context, processing a single sequence at position 1000. The bias for query position 1000 attending to key position 500 is -m_h × 500. For the steepest slope head (say m_h = 1/2), that's a -250 bias subtracted from a raw score that typically sits in the [-3, 3] range. Softmax pushes that pair's weight to essentially zero. For the gentlest slope head (say m_h = 1/256), the bias is -1.95, a real penalty but not enough to fully suppress the pair.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | ALiBi | RoPE | Learned PE |
|---|---|---|---|
| Where applied | Pre-softmax score bias | Rotation on Q, K | Added to input embedding |
| Input embedding modified? | No | No | Yes (additive) |
| Trainable parameters | None (slopes are fixed) | None | One vector per position |
| Length extrapolation | Strong (smooth at any distance) | Weak; needs PI/YaRN | None (hard max cap) |
| Adoption (2024) | BLOOM, MPT | Llama, Mistral, Qwen, DeepSeek | Original BERT, GPT-2 |
Real products, models, and research that use this idea.
- BLOOM (BigScience) used ALiBi.
- MPT-7B and MPT-30B (MosaicML) used ALiBi for cheap context extension at inference.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the bias added BEFORE softmax, not after?
Pre-softmax additive bias converts to multiplicative weighting (via the exponential) on the attention probabilities. Post-softmax addition would break the probability simplex (sum to one) and not produce the desired exponential locality bias.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying ALiBi modifies softmax or operates after it, ALiBi is purely a pre-softmax score bias. Also: saying ALiBi uses a positional embedding on the input, it doesn't, that's the whole point.
60 second bullets to scan on the way to the call.
Pre-softmax score bias formula and where it enters
Per-head slopes as a geometric series across heads
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.