Define T5's relative position bias and what bucketing buys you

Short answer·Medium·4.0 · 0·~3 min

Attempt it

T5 injects positional information via a relative position bias added to attention scores. Describe the mechanism precisely, including what 'bucketing' means and why T5 uses it. Note where parameters are shared and where they are not.

Free · 2 AI evals / day

T5 encodes position differently from BERT or GPT. There is no learned positional embedding added to the input tokens, and no sinusoidal pattern injected anywhere. Instead, the position signal lands on the attention score itself. For each query position i and key position j, T5 adds a learned scalar bias to the pre-softmax score: score(i, j) = (q_i dot k_j) / sqrt(d_k) + b(i - j, head) The bias depends only on the offset (i - j), not on the absolute positions. Two pairs of tokens at distance 5 in the sequence get the same bias, regardless of where in the sequence they appear. The practical question is: how do you store a learned scalar for every possible offset without exploding parameter counts? T5 uses bucketing. Offsets are mapped through a function that puts each small offset in its own bucket and groups larger offsets into log-spaced buckets. A typical T5 config has 32 buckets covering offsets up to 128, with everything beyond 128 collapsing into the same bucket. The full table is a small matrix of shape (num_heads x num_buckets), shared across all layers. Different heads get different rows, so each head can learn its own distance preference.

**T5 puts the position signal on the score side.** No input embedding, no Q/K rotation, just a learned scalar added to the pre-softmax attention score for each (query, key) pair, indexed by the offset between positions. The formula is straightforward: $$\text{score}(i, j) = \frac{q_i \cdot k_j}{\sqrt{d_k}} + b(i - j, h)$$ Where `b` is the learned bias table and `h` is the head index. The bias depends only on the relative offset (i - j), which is what makes it a relative position encoding rather than an absolute one. Bucketing is the parameter-saving trick. A function maps each offset to one of a small fixed number of buckets, typically 32. Small offsets like 0, 1, 2, 3, 4 each get unique buckets so close-range structure is captured precisely. Larger offsets are log-spaced (offsets 8-15 share a bucket, 16-31 share another, etc.) so the parameter count stays small while still distinguishing 'somewhat far' from 'very far'. Offsets beyond a max distance (often 128) all map to the same final bucket, encoding very long distance relations coarsely but cheaply. The sharing pattern matters: the bias table is shared across ALL transformer layers, but each head gets its own row. So total bias parameters are about (num_heads x num_buckets), a few thousand floats for a typical config, the same set reused at every layer. The contrast with RoPE is clean: T5 adds learned biases to the score; RoPE rotates Q and K vectors so the relative property emerges from the rotation algebra rather than from learned parameters.

Quick reference

TL;DR

T5 adds a learned scalar bias to the pre-softmax score, indexed by the offset (i minus j) through a log-spaced bucketing function so the parameter count stays small.

Memory aid

Easy to grasp

Think of a long auditorium with numbered seats. Instead of giving each seat its own name tag, T5 hands out a small set of distance stickers: one for 'right next to me', one for 'a few seats away', one for 'across the row', one for 'somewhere far'. Two people sitting close get the same close-distance sticker no matter where they are in the auditorium; two people far apart get the same far-distance sticker. The model learns one rule per sticker about how much to listen, and that rule applies everywhere. The auditorium can be any size and the model still works, because the stickers cover the whole range with a small fixed set.

Key concepts

T5 model Relative position encoding Positional encoding

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6m: score-side bias formula, bucketing function with log-spaced buckets, parameter accounting and layer sharing, contrast with RoPE and ALiBi, length-extrapolation limits, and interaction with FlashAttention.

Code example

python

import torch

def relative_position_bucket(relative_position, num_buckets=32, max_distance=128):
    # T5's bucketing function: half the buckets for negative offsets,
    # half for positive; close offsets get unique buckets,
    # far offsets get log-spaced buckets.
    ret = 0
    n = -relative_position
    num_buckets //= 2
    ret += (n < 0).long() * num_buckets
    n = n.abs()
    max_exact = num_buckets // 2
    is_small = n < max_exact
    val_if_large = max_exact + (
        torch.log(n.float() / max_exact)
        / torch.log(torch.tensor(max_distance / max_exact))
        * (num_buckets - max_exact)
    ).long()
    val_if_large = torch.min(val_if_large, torch.full_like(val_if_large, num_buckets - 1))
    ret += torch.where(is_small, n, val_if_large)
    return ret

# bias table: (num_heads, num_buckets), shared across all layers
bias_table = torch.nn.Parameter(torch.zeros(12, 32))

# at attention time: compute offsets, bucket them, lookup
i = torch.arange(seq_len)[:, None]
j = torch.arange(seq_len)[None, :]
buckets = relative_position_bucket(j - i)
bias = bias_table[:, buckets]  # (num_heads, seq_len, seq_len)
scores = (q @ k.transpose(-2, -1)) / math.sqrt(d_k) + bias

Real-world examples

Real products, models, and research that use this idea.

T5 (Raffel et al. 2020) and its descendants (Flan-T5, mT5) all use this bias scheme as the only position signal.
Google's UL2 follows T5's relative bias design for its denoising-objective pretraining.
ByT5 (byte-level T5) keeps the same bucketing function despite operating on much longer character sequences.
Most 2026 frontier decoders (Llama 4 Maverick, Claude Opus 4.7, Gemini 3.1 Pro, DeepSeek V4) have moved away from T5-style biases to RoPE plus YaRN or position interpolation for long-context training.
Swin Transformer for vision uses a related learned bias table idea inside windows, drawing directly on the T5 design.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy log-spaced buckets specifically, instead of uniform or quadratic spacing?

QCan T5 bias extrapolate to sequences longer than what it was trained on?

QHow does the bias interact with FlashAttention and other fused kernels?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing T5's bias with a learned absolute positional embedding added to the input. The bias goes onto the score, not the embedding, and depends only on the offset between positions, not on absolute index.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.