Click any words you think contain an error. Click again to unmark.
With left padding, pads sit before the real tokens, so the causal mask lets real tokens attend to them. Apply the padding mask with the causal mask before softmax. Loss masking does not fix this.
Picture a study group where some students arrived late, so the first pages of their notebooks are blank. The class rule is 'only look at notes from earlier in the lecture' (causal mask). That rule says nothing about skipping blank pages. So a late student who reviews their own notes reads the blank early pages too and tries to study from them. You need two rules together: 'only look at earlier notes' AND 'skip blank pages'. Left padding with only the causal mask applies one rule when you need both, and the model trains on blank-paper context.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The interaction between causal and padding masks causes a subtle bug in transformer training. The argument 'the causal mask already restricts attention to the past, so the padding mask is redundant' sounds right. It fails with left padding, where the pads sit in the past of every real token. The two masks block orthogonal sets of positions and neither subsumes the other.
Mental model: masks compose by intersection of legal keys. Causal says 'past only', padding says 'real only'. Their intersection is 'past AND real', which is what every query needs.
The orthogonality of causal and padding masks
A mask is a specification of which keys are legal for each query. Two masks compose by taking the intersection of their legal-key sets.
The causal mask
Causal mask: for query at position t, legal keys are at positions 0 through t-1 (or 0 through t if self-attention is allowed). The illegal set is positions t+1 through T-1: the future.
The padding mask
Padding mask: for any query, legal keys are at non-pad positions. The illegal set is the pad columns, which depend on the sequence's real length and are independent of the query's position.
Why neither subsumes the other
Consider the four cases for a real query in a padded batch:
- Real query, real key in the past. Causal allows (key is past), padding allows (key is real). Both masks legal. Intersection: legal.
- Real query, pad key in the past. Causal allows (key is in past columns), padding blocks (key is pad). Intersection: illegal. Only the padding mask catches this.
- Real query, real key in the future. Causal blocks (key is future), padding allows (key is real). Intersection: illegal. Only the causal mask catches this.
- Real query, pad key in the future. Both block. Intersection: illegal.
The second and third rows are the key cases. The causal mask alone misses the second; the padding mask alone misses the third.
Left padding makes the second row common
In left padding, the pads sit at the start of each short sequence. Consider a sequence with 5 pad tokens (positions 0-4) followed by 3 real tokens (positions 5, 6, 7).
Position 5 (the first real token) attends to positions 0-5 under the causal mask. Five of those six keys are pads. Without a padding mask, its attention output is mostly pad content. Positions 6 and 7 are the same. The contamination is at real positions, whose outputs feed the next layer, the KV cache, and the loss.
Why right padding hides the bug
In right padding, the pads sit at the end: 3 real tokens (positions 0, 1, 2), then 5 pads (positions 3-7). Every real query's past holds only real tokens, so the causal mask alone keeps real outputs clean at every layer. Only the pad rows mix in pad content, and the loss mask ignores them. The second row of the table never occurs for a real query. This is why code that drops the padding mask can look correct with right padding. It then breaks when the batch is switched to left padding.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import torch
import torch.nn.functional as F
B, T, n_heads, d_head = 2, 8, 4, 16
real_lengths = torch.tensor([3, 8]) # Seq 0: 5 pad + 3 real (left padded). Seq 1: all real.
Q = torch.randn(B, n_heads, T, d_head)
K = torch.randn(B, n_heads, T, d_head)
V = torch.randn(B, n_heads, T, d_head)
# Padding mask: very negative at pad columns, 0 at real columns. Shape (B, 1, 1, T).
# Left padding: the real tokens are the last real_lengths positions.
is_real = torch.arange(T)[None, :] >= (T - real_lengths)[:, None] # (B, T)
# A finite minimum (not -inf) keeps fully masked pad rows from turning into NaN.
neg = torch.finfo(Q.dtype).min
pad_mask = torch.where(is_real, 0.0, neg)[:, None, None, :]
# Causal mask: -inf above the diagonal, 0 on and below. Shape (1, 1, T, T).
causal_mask = torch.triu(torch.full((T, T), float('-inf')), diagonal=1)
causal_mask = causal_mask[None, None, :, :]
# Combine BOTH masks before softmax. This is the correct approach.
scores = Q @ K.transpose(-2, -1) / (d_head ** 0.5)
scores = scores + causal_mask + pad_mask # any position future OR pad is blocked
weights = F.softmax(scores, dim=-1)
output = weights @ V
# Loss masking is a SEPARATE step on the LM head output, not a replacement
# for the attention-level padding mask.Real products, models, and research that use this idea.
- PyTorch's torch.nn.MultiheadAttention takes both attn_mask (causal) and key_padding_mask separately because they compose.
- Hugging Face Transformers' models combine the attention_mask (padding) with an internally constructed causal mask in decoder layers. Its docs recommend left padding for batched generation with decoder-only models, where this combination matters.
What an interviewer would ask next. Try answering before peeking at the approach.
QWould switching to right padding fix the bug?
For real tokens, yes. With right padding, every pad comes after every real token, so the causal mask alone already hides pads from real queries. The loss mask ignores the pad rows. It is fragile, though. Batched generation needs left padding, and packing or bidirectional attention bring the problem back, so keeping the padding mask is the safe default.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing the causal mask subsumes the padding mask because 'pad tokens are in the past'. The causal mask permits the past; it does not exclude padding from the past.
60 second bullets to scan on the way to the call.
Why the causal mask alone is insufficient for left padded batches
Why the padding mask alone is insufficient for causal-LM training
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.