A batch packs a causal LM with right padded sequences and applies only the causal mask. Spot the mistake.

Spot the error·Medium·4.0 · 0·~2 min·Asked atAi4bharat Harvey Persistent·Relevant atMicrosoft

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

**The engineer's argument is wrong** because the causal mask and the padding mask block different things, and neither implies the other. The **causal mask** says: 'every query can attend to positions 0 through t-1, never to the future'. It is lower-triangular. It permits the past. The **padding mask** says: 'no query can attend to positions that are pad tokens'. It blocks specific columns regardless of their time index. In a right-padded batch, the pad tokens sit at the end of each short sequence. Consider a sequence with 3 real tokens followed by 5 pad tokens, batched alongside a sequence with 8 real tokens. The causal mask alone permits the 3rd real token to attend to all earlier positions, but in this short sequence those earlier positions are entirely real, so this case looks fine. The bug shows up when you consider what happens at the pad positions themselves. Position 4 (the first pad) is allowed to attend to positions 0-3 under the causal mask. That is fine because position 4 is going to be loss-masked anyway and its output does not matter. But position 5 (the second pad) is also allowed to attend backward... and position 7 is the same. The model still computes attention at every pad position, contaminates the K, V cache, and any code path that uses representations at pad positions reads contaminated data. **Loss masking** does not fix this because loss masking handles outputs, not inputs. The contamination has already happened in the attention layer. The fix is to apply the padding mask alongside the causal mask: combine them via logical AND (or sum the -inf masks) before softmax.

**The causal mask and the padding mask are orthogonal constraints,** not redundant. Each blocks something the other does not, so removing either one leaves a hole. - **Causal mask** blocks attention to **future** positions. Lower-triangular pattern over (time x time). - **Padding mask** blocks attention to **pad** positions. Column-based pattern over (any-time x pad-columns). In a right-padded batch with a short sequence, the causal mask permits attention from position t to positions 0..t-1. Some of those earlier positions may themselves be pad if the sequence is shorter than t suggests, but in right-padding the pads come after the reals, so this particular ordering of pads relative to reals does not immediately bite. **Where it does bite:** every position, including pad positions, still has its attention computed during training. The pad positions' K and V are mixed into the weighted sum at every later query position that attends to them under the causal mask. Even worse, in many implementations, the model is asked to predict next tokens at every position, including pad positions; the loss is then masked to ignore those, but the attention contamination has already corrupted the activations that produced those predictions. **The fix.** Combine the masks before softmax: ```python additive = causal_mask + padding_mask # both -inf where blocked scores = scores + additive weights = softmax(scores, dim=-1) ``` Any position that is either future or padding gets -inf and contributes nothing to the softmax. Loss masking is still needed to handle the prediction layer correctly, but it is a separate fix from the attention-level masking. **Why the engineer's intuition is tempting but wrong.** 'Pad tokens are in the past' is true, but the causal mask permits the past; it does not exclude any subset of the past. The causal mask is one-sided in time; the padding mask is one-sided in token-identity. Neither subsumes the other.

TL;DR

Causal mask blocks futures, padding mask blocks pad. Neither is a superset of the other. With right-padding you need both, applied together before softmax. Loss masking on outputs does not fix attention contamination.

Memory aid

Easy to grasp

Picture a study group where some students arrived late and have empty notebook pages for the early part of the lecture. The class rule is 'only look at notes from earlier in the lecture' (causal mask). That rule says nothing about ignoring the empty pages, so a student trying to review their own notes from later in the lecture will happily read the empty pages from the late-arriving student next to them and try to study from blank paper. You need two rules together: 'only look at earlier notes' AND 'skip empty pages'. Right-padding plus causal mask only is one rule when you need both, and the model trains on blank-paper context.

Key concepts

Causal mask Padding mask Mask combination Loss masking

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The interaction between causal and padding masks is the source of one of the most subtle bugs in transformer training. The argument 'the causal mask already restricts attention to the past, so the padding mask is redundant' is intuitively appealing and operationally wrong. The two masks block orthogonal sets of positions and neither subsumes the other.

This deep dive walks the orthogonality argument, demonstrates the bug with a worked example, explains why loss masking is a separate (and insufficient) fix, surveys the production-grade alternatives that avoid the composition issue entirely, and provides a unit-test recipe that catches this class of bug in CI.

Mental model: masks compose by intersection of legal keys. Causal says 'past only', padding says 'real only'. Their intersection is 'past AND real', which is what every query needs.

The orthogonality of causal and padding masks

A mask is a specification of which keys are legal for each query. Two masks compose by taking the intersection of their legal-key sets.

The causal mask

Causal mask: for query at position t, legal keys are at positions 0 through t-1 (or 0 through t if self-attention is allowed). The illegal set is positions t+1 through T-1: the future.

The padding mask

Padding mask: for any query, legal keys are at non-pad positions. The illegal set is the pad columns, which depend on the sequence's real length and are independent of the query's position.

Why neither subsumes the other

Consider two specific examples in a right-padded batch:

Real query, real key in the past. Causal allows (key is past), padding allows (key is real). Both masks legal. Intersection: legal.
Real query, pad key in the past. Causal allows (key is in past columns), padding blocks (key is pad). Intersection: illegal. Only the padding mask catches this.
Real query, real key in the future. Causal blocks (key is future), padding allows (key is real). Intersection: illegal. Only the causal mask catches this.
Real query, pad key in the future. Both block. Intersection: illegal.

The second and third rows are the load-bearing cases. The causal mask alone misses the second; the padding mask alone misses the third.

Right-padding makes the second row common

In right-padding, the pads sit at the end of each short sequence. Consider a sequence with 3 real tokens (positions 0, 1, 2) followed by 5 pad tokens (positions 3-7). At training time, every position has its attention computed, including positions 3-7.

Position 5 (a pad) attends to positions 0-4 under the causal mask. Positions 0, 1, 2 are real; positions 3, 4 are pad. Without a padding mask, position 5's attention output is contaminated by other pad positions in its 'past'. The contamination is not at the loss layer (position 5's output is loss-masked) but at the activations that feed forward through residual streams and into later layers and into the KV cache.

If position 5 is part of a multi-layer transformer, its layer-1 contaminated output becomes layer-2's input, which means layer-2's attention at every position can read from layer-1-contaminated representations of pad positions. The contamination propagates through depth.

Worked example of the bug

Why loss masking is necessary but not sufficient

Production-grade alternatives and a unit-test recipe

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import torch
import torch.nn.functional as F

B, T, n_heads, d_head = 2, 8, 4, 16
real_lengths = torch.tensor([3, 8])  # Seq 0: 3 real + 5 pad. Seq 1: all real.

Q = torch.randn(B, n_heads, T, d_head)
K = torch.randn(B, n_heads, T, d_head)
V = torch.randn(B, n_heads, T, d_head)

# Padding mask: -inf at pad columns, 0 at real columns. Shape (B, 1, 1, T).
is_real = torch.arange(T)[None, :] < real_lengths[:, None]  # (B, T)
pad_mask = torch.where(is_real, 0.0, float('-inf'))[:, None, None, :]

# Causal mask: -inf above the diagonal, 0 on and below. Shape (1, 1, T, T).
causal_mask = torch.triu(torch.full((T, T), float('-inf')), diagonal=1)
causal_mask = causal_mask[None, None, :, :]

# Combine BOTH masks before softmax. This is the correct approach.
scores = Q @ K.transpose(-2, -1) / (d_head ** 0.5)
scores = scores + causal_mask + pad_mask  # any position future OR pad becomes -inf
weights = F.softmax(scores, dim=-1)
output = weights @ V

# Loss masking is a SEPARATE step on the LM head output, not a replacement
# for the attention-level padding mask.

Real products, models, and research that use this idea.

PyTorch's torch.nn.MultiheadAttention takes both attn_mask (causal) and key_padding_mask separately because they compose.
Hugging Face Transformers' models combine the attention_mask (padding) with an internally constructed causal mask in decoder layers.

What an interviewer would ask next. Try answering before peeking at the approach.

QWould left-padding instead of right-padding fix the bug?

No. Left-padding puts pads at the start of each short sequence; real tokens at later positions still attend backward into the pads under the causal mask. The combinatorial structure is symmetric: the padding mask is required regardless of whether padding is on the left or the right.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing the causal mask subsumes the padding mask because 'pad tokens are in the past'. The causal mask permits the past; it does not exclude padding from the past.

60 second bullets to scan on the way to the call.

Why the causal mask alone is insufficient for variable-length right-padded batches
Why the padding mask alone is insufficient for causal-LM training

Primary sources. Browse if you want the original framing.