ALiBi, decode the acronym and pinpoint where in the pipeline it injects position

Attempt it

ALiBi stands for Attention with Linear Biases. Unlike RoPE or sinusoidal PE, ALiBi adds no positional information to the input embeddings or to Q and K. Instead, it adds a per head linear distance bias directly to the pre softmax attention score matrix: score_ij gets penalized by -m_h * |i - j|, where m_h is a fixed per head slope from a geometric series. Heads with steep slopes attend locally; heads with shallow slopes attend long range. The bias is purely additive, applied right before softmax, costs no extra parameters, and extrapolates to longer than training sequences more gracefully than learned PE.

Your answer

Quick reference

TL;DR

ALiBi = Attention with Linear Biases. It adds a per head linear distance penalty to pre softmax scores; no input position embedding.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a group chat where every person hears every other person, but each listener wears a different pair of headphones. Some headphones get louder the closer the speaker sits; others barely change with distance. The chat itself has no name tags, no seating chart, no announcement of who is two seats away from whom. The volume curves are the only source of position information. ALiBi is that volume curve applied to attention: instead of telling tokens where they sit, it just turns down the volume on far away tokens by a fixed amount that grows with distance. Different attention heads use different volume curves, so some heads focus on nearby tokens and others still hear the whole room.

Key concepts

Positional encoding ALiBi Length extrapolation Attention bias

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

6-8 min: ALiBi acronym + injection point + formula + slope schedule + length extrapolation property + comparison to RoPE and PE + production adoption.

Real-world examples

Real products, models, and research that use this idea.

BLOOM 176B (BigScience, 2022) shipped with ALiBi as its positional scheme.
MPT-7B and MPT-30B (MosaicML, 2023) used ALiBi and advertised length extrapolation past training context.
Modern frontier LLMs (Llama 4 Maverick, Mistral, Qwen 3.5, DeepSeek V4) converged on RoPE with PI or YaRN rather than ALiBi.
Several efficient serving research stacks still favor ALiBi because the closed form bias composes well with FlashAttention kernels.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy did frontier labs move from ALiBi to RoPE plus extension tricks?

Sign in or go Pro to see the approach for this follow-up.

QHow does ALiBi interact with FlashAttention?

Sign in or go Pro to see the approach for this follow-up.

QWhat goes wrong if you use a single shared slope across all heads instead of a geometric series?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying ALiBi modifies the input embeddings or the Q/K vectors. It does neither. The bias lives at the score matrix, one line after the QK^T multiplication.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Press et al., Train Short, Test Long: Attention with Linear Biases (ALiBi)