How does ALiBi inject position information without using positional embeddings?
ALiBi injects position by penalizing far apart token pairs before softmax, with each head set to a different decay rate.
Imagine a room of listeners who all turn the volume down on speakers sitting far away. The further a speaker sits, the quieter they sound. Each listener in the room has different ears: some can only hear nearby voices, while others still catch words from across the room. The model never gets a 'seat number' for any word. Instead, the volume drop quietly tells the model who is close and who is far. And because listeners have such different hearing, some end up paying attention to neighbors only, while others can listen across the whole room. The room handles short and long range together, just from how loud each speaker sounds.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Walk the formula, explain the geometric per-head slopes, articulate the extrapolation argument (smooth at any distance), contrast with RoPE/learned PE, name BLOOM and MPT as adopters, and note that frontier LLMs converged on RoPE.
| Property | ALiBi | RoPE | Learned PE |
|---|---|---|---|
| Where applied | Pre-softmax score bias | Rotation on Q, K | Added to input embedding |
| Input embedding modified? | No | No | Yes (additive) |
| Trainable parameters | None (slopes are fixed) | None | One vector per position |
| Length extrapolation | Strong (smooth at any distance) | Weak; needs PI/YaRN | None (hard max cap) |
| Adoption (2024) | BLOOM, MPT | Llama, Mistral, Qwen, DeepSeek | Original BERT, GPT-2 |
Real products, models, and research that use this idea.
- BLOOM (BigScience) used ALiBi.
- MPT-7B and MPT-30B (MosaicML) used ALiBi for cheap context extension at inference.
- Cerebras-GPT and several academic long context models used ALiBi during the 2022-2023 long context wave.
- Press et al.'s original paper trained 1k-context models and showed perplexity stability out to 2k-16k at inference.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the bias added BEFORE softmax, not after?
QWalk through why ALiBi extrapolates and where the claim has limits.
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying ALiBi modifies softmax or operates after it, ALiBi is purely a pre-softmax score bias. Also: saying ALiBi uses a positional embedding on the input, it doesn't, that's the whole point.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.