ALiBi, decode the acronym and pinpoint where in the pipeline it injects position

**ALiBi** stands for Attention with Linear Biases. It is a way to give a transformer a sense of token position without adding any positional vector to the input. Standard schemes like sinusoidal PE or learned PE add a position vector to each token embedding before attention runs. RoPE rotates the Q and K vectors instead. ALiBi does neither. It leaves the embeddings, Q, and K alone. The injection happens at the score matrix. After the model computes `scores = QK^T / sqrt(d_k)`, ALiBi adds a per head bias matrix where each entry is `-m_h * |i - j|`. Here i and j are query and key positions, and m_h is a fixed slope chosen per head from a geometric series like 1/2, 1/4, 1/8. Because the penalty grows linearly with distance, far away tokens get pushed down before softmax. Heads with steep slopes effectively look at a local window; heads with shallow slopes still see the whole sequence. The scheme adds no learnable parameters and works at any sequence length.

**ALiBi** expands to Attention with Linear Biases. The injection point is the score matrix, not the embedding and not the Q/K projections. The modification is one line. After computing `scores = QK^T / sqrt(d_k)`, ALiBi adds a per head, fixed bias: $$s_{ij} \mathrel{+}= -m_h \cdot \lvert i - j \rvert$$ The slope m_h is per head and fixed (not learned). It is drawn from a geometric series so different heads attend at different ranges. There is no learned table of positions and no rotation of the Q or K vectors. Three things follow. First, ALiBi adds zero parameters. Second, the bias function is well defined at any distance, which gives the scheme its length-extrapolation property, you can run a 4k-trained model on 16k inputs without retraining or interpolation tricks. Third, the implementation is trivial: every attention kernel already supports an additive bias matrix B in `scores += B`, so dropping ALiBi into existing code is a single tensor add. It was widely used in BLOOM and MPT and was a serious contender for length extrapolation in 2022. Modern frontier LLMs have mostly converged on RoPE with extension tricks (PI, NTK-aware, YaRN) instead.