Define T5's relative position bias and what bucketing buys you

Short answer·Medium·4.0 · 0·~3 min·Asked atNykaa Observe Ai Robinhood

Attempt it

T5 injects positional information via a relative position bias added to attention scores. Describe the mechanism precisely, including what 'bucketing' means and why T5 uses it. Note where parameters are shared and where they are not.

Free · 2 AI evals / day

T5 encodes position differently from BERT or GPT. There is no learned positional embedding added to the input tokens, and no sinusoidal pattern injected anywhere. Instead, the position signal lands on the attention score itself. For each query position i and key position j, T5 adds a learned scalar bias to the pre-softmax score: score(i, j) = (q_i dot k_j) / sqrt(d_k) + b(i - j, head) The bias depends only on the offset (i - j), not on the absolute positions. Two pairs of tokens at distance 5 in the sequence get the same bias, regardless of where in the sequence they appear. The practical question is: how do you store a learned scalar for every possible offset without exploding parameter counts? T5 uses bucketing. Offsets are mapped through a function that puts each small offset in its own bucket and groups larger offsets into log-spaced buckets. A typical T5 config has 32 buckets covering offsets up to 128, with everything beyond 128 collapsing into the same bucket. The full table is a small matrix of shape (num_heads x num_buckets), shared across all layers. Different heads get different rows, so each head can learn its own distance preference.

**T5 puts the position signal on the score side.** No input embedding, no Q/K rotation, just a learned scalar added to the pre-softmax attention score for each (query, key) pair, indexed by the offset between positions. The formula is straightforward: $$\text{score}(i, j) = \frac{q_i \cdot k_j}{\sqrt{d_k}} + b(i - j, h)$$ Where `b` is the learned bias table and `h` is the head index. The bias depends only on the relative offset (i - j), which is what makes it a relative position encoding rather than an absolute one. Bucketing is the parameter-saving trick. A function maps each offset to one of a small fixed number of buckets, typically 32. Small offsets like 0, 1, 2, 3, 4 each get unique buckets so close-range structure is captured precisely. Larger offsets are log-spaced (offsets 8-15 share a bucket, 16-31 share another, etc.) so the parameter count stays small while still distinguishing 'somewhat far' from 'very far'. Offsets beyond a max distance (often 128) all map to the same final bucket, encoding very long distance relations coarsely but cheaply. The sharing pattern matters: the bias table is shared across ALL transformer layers, but each head gets its own row. So total bias parameters are about (num_heads x num_buckets), a few thousand floats for a typical config, the same set reused at every layer. The contrast with RoPE is clean: T5 adds learned biases to the score; RoPE rotates Q and K vectors so the relative property emerges from the rotation algebra rather than from learned parameters.

TL;DR

T5 adds a learned scalar bias to the pre-softmax score, indexed by the offset (i minus j) through a log-spaced bucketing function so the parameter count stays small.

Memory aid

Easy to grasp

Think of a long auditorium with numbered seats. Instead of giving each seat its own name tag, T5 hands out a small set of distance stickers: one for 'right next to me', one for 'a few seats away', one for 'across the row', one for 'somewhere far'. Two people sitting close get the same close-distance sticker no matter where they are in the auditorium; two people far apart get the same far-distance sticker. The model learns one rule per sticker about how much to listen, and that rule applies everywhere. The auditorium can be any size and the model still works, because the stickers cover the whole range with a small fixed set.

Key concepts

T5 model Relative position encoding Positional encoding

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

T5's relative position bias is the cleanest example of the 'put position on the score' design philosophy. Instead of injecting position into the input embeddings (like BERT's learned PE or the original Transformer's sinusoidal PE) or into the Q and K vectors (like RoPE), T5 adds a learned scalar to the pre-softmax attention score for each pair of positions. The scalar depends only on the offset between positions, making it a relative encoding.

The scheme is elegant for two reasons. It's translation-invariant by construction: shifting the whole sequence by k positions changes nothing because the bias depends only on offsets. And it's parameter-efficient: a small bucketing function maps arbitrary offsets to a fixed set of buckets, so total bias parameters stay in the low thousands regardless of context length.

This deep dive walks through the bias formula, the bucketing function in detail, the sharing pattern across layers and heads, the limits of length extrapolation, and the trade-offs vs RoPE and ALiBi.

The bias formula and where it sits

The attention score in T5 is the standard dot product plus a learned offset-indexed bias:

\text{score}(i, j, h) = \frac{q_i \cdot k_j}{\sqrt{d_k}} + b(i - j, h)

Where i is the query position, j is the key position, h is the head index, and b is the learned bias table.

What sits where

Input embeddings: no positional component; just token embeddings.
Q and K vectors: no rotation, no positional mixing. Pure projections of the hidden states.
Score matrix: bias added here, just before softmax.

The contrast with other schemes is sharp. BERT adds a learned PE vector to the input embeddings. The original Transformer adds sinusoidal PE to the input. RoPE rotates Q and K by position-dependent angles inside the attention. T5 does none of these; the only position signal is the score-side bias.

Translation invariance for free

Because the bias depends only on i - j, shifting all positions by k leaves every bias value unchanged. This is the defining property of relative position encoding: the model cares about distances between tokens, not their absolute indices.

The bucketing function

Sharing pattern and parameter accounting

Length extrapolation and the comparison to RoPE / ALiBi

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import torch

def relative_position_bucket(relative_position, num_buckets=32, max_distance=128):
    # T5's bucketing function: half the buckets for negative offsets,
    # half for positive; close offsets get unique buckets,
    # far offsets get log-spaced buckets.
    ret = 0
    n = -relative_position
    num_buckets //= 2
    ret += (n < 0).long() * num_buckets
    n = n.abs()
    max_exact = num_buckets // 2
    is_small = n < max_exact
    val_if_large = max_exact + (
        torch.log(n.float() / max_exact)
        / torch.log(torch.tensor(max_distance / max_exact))
        * (num_buckets - max_exact)
    ).long()
    val_if_large = torch.min(val_if_large, torch.full_like(val_if_large, num_buckets - 1))
    ret += torch.where(is_small, n, val_if_large)
    return ret

# bias table: (num_heads, num_buckets), shared across all layers
bias_table = torch.nn.Parameter(torch.zeros(12, 32))

# at attention time: compute offsets, bucket them, lookup
i = torch.arange(seq_len)[:, None]
j = torch.arange(seq_len)[None, :]
buckets = relative_position_bucket(j - i)
bias = bias_table[:, buckets]  # (num_heads, seq_len, seq_len)
scores = (q @ k.transpose(-2, -1)) / math.sqrt(d_k) + bias

Real products, models, and research that use this idea.

T5 (Raffel et al. 2020) and its descendants (Flan-T5, mT5) all use this bias scheme as the only position signal.
Google's UL2 follows T5's relative bias design for its denoising-objective pretraining.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy log-spaced buckets specifically, instead of uniform or quadratic spacing?

Linguistic and visual structure tends to fall off in importance roughly logarithmically with distance: very close tokens carry most syntactic dependency, medium-range tokens carry phrase-level structure, and very far tokens carry coarse topical context. Log spacing matches that distribution by giving close offsets fine resolution and far offsets coarse resolution, getting more useful inductive bias per parameter.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing T5's bias with a learned absolute positional embedding added to the input. The bias goes onto the score, not the embedding, and depends only on the offset between positions, not on absolute index.

60 second bullets to scan on the way to the call.

Where T5 puts the position signal in the attention pipeline
Exact formula for the modified attention score with bias

Primary sources. Browse if you want the original framing.