How does ALiBi inject position information without using positional embeddings?

Attempt it

Quick reference

TL;DR

ALiBi injects position by penalizing far apart token pairs before softmax, with each head set to a different decay rate.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a room of listeners who all turn the volume down on speakers sitting far away. The further a speaker sits, the quieter they sound. Each listener in the room has different ears: some can only hear nearby voices, while others still catch words from across the room. The model never gets a 'seat number' for any word. Instead, the volume drop quietly tells the model who is close and who is far. And because listeners have such different hearing, some end up paying attention to neighbors only, while others can listen across the whole room. The room handles short and long range together, just from how loud each speaker sounds.

Key concepts

Attention with Linear Biases Positional encoding

Quick scan · 2–4 min

Everything important, quickly.

Walk the formula, explain the geometric per-head slopes, articulate the extrapolation argument (smooth at any distance), contrast with RoPE/learned PE, name BLOOM and MPT as adopters, and note that frontier LLMs converged on RoPE.

Side-by-side

Property	ALiBi	RoPE	Learned PE
Where applied	Pre-softmax score bias	Rotation on Q, K	Added to input embedding
Input embedding modified?	No	No	Yes (additive)
Trainable parameters	None (slopes are fixed)	None	One vector per position
Length extrapolation	Strong (smooth at any distance)	Weak; needs PI/YaRN	None (hard max cap)
Adoption (2024)	BLOOM, MPT	Llama, Mistral, Qwen, DeepSeek	Original BERT, GPT-2

Real-world examples

Real products, models, and research that use this idea.

BLOOM (BigScience) used ALiBi.
MPT-7B and MPT-30B (MosaicML) used ALiBi for cheap context extension at inference.
Cerebras-GPT and several academic long context models used ALiBi during the 2022-2023 long context wave.
Press et al.'s original paper trained 1k-context models and showed perplexity stability out to 2k-16k at inference.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the bias added BEFORE softmax, not after?

Sign in or go Pro to see the approach for this follow-up.

QWalk through why ALiBi extrapolates and where the claim has limits.

Sign in or go Pro to see the approach for this follow-up.

QWhy have modern frontier LLMs converged on RoPE rather than ALiBi?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.