A causal attention mask is built for a sequence of length 4, using 0 indexed rows and columns. The convention is that allowed positions hold 0 and disallowed positions hold negative infinity, and the mask is added to the score matrix before softmax. Predict the value stored at row index 2 (the third query token), column index 3 (the fourth key token).
Position (2, 3) is strictly above the diagonal, so the causal mask holds -inf there and softmax zeros that attention weight after addition.
Imagine reading a book one word at a time and you are only allowed to look at words you have already read. Word number 3 (in zero-indexed counting, the third one) can look back at words 0, 1, and 2 (itself). It cannot peek ahead at word 4 because you have not read it yet. The causal mask is the rule that enforces 'no peeking ahead'. It works by writing a giant negative number on every forbidden position before the model decides where to look. After the model turns those numbers into voting weights, a giant negative number becomes essentially zero votes, so the forbidden word contributes nothing to what gets read next.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The causal mask is the simplest piece of machinery that turns a bidirectional attention layer into an autoregressive one. It enforces one rule: a query at position i may only attend to keys at positions j <= i. Without it, a language model trained to predict the next token could cheat by looking at the answer.
This question probes whether the candidate knows the row/column semantics, the additive 0-vs-(-inf) convention, the diagonal-allowed rule, and how it composes with softmax. All four are routinely confused, especially the diagonal rule (off by one) and the row/column meaning (which is the attender, which is the target).
This deep dive walks through the matrix structure, the softmax interaction, how production kernels handle the mask without materializing it, and why decode-time attention skips the mask entirely.
Row attends to column
In attention, the score matrix S = QK^T / sqrt(d_k) has shape (T, T) where:
- Row index
icorresponds to the query position. This is the token doing the attending. - Column index
jcorresponds to the key position. This is the token being attended to.
The attention weight a_ij = softmax_j(S)_ij answers the question: "how much does token i attend to token j?"
What 'causal' means
In autoregressive generation, token i is produced after tokens 0, 1, ..., i-1 and may use them as input. It is also allowed to use itself (the residual stream is built up at each position). It may NOT use tokens i+1, i+2, ..., T-1, those are future tokens.
The constraint in matrix terms: allow column j for row i only when j <= i.
Question entry (2, 3)
Row i = 2, column j = 3. We have j > i (3 > 2), so this is a forbidden entry, the mask blocks it with -inf.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import torch
# Build the 4x4 causal mask used at attention.
# 0 = allowed, -inf = forbidden.
T = 4
mask = torch.triu(torch.full((T, T), float('-inf')), diagonal=1)
print(mask)
# tensor([[0., -inf, -inf, -inf],
# [0., 0., -inf, -inf],
# [0., 0., 0., -inf],
# [0., 0., 0., 0.]])
print(mask[2, 3]) # tensor(-inf)
# Applied to scores before softmax
scores = torch.randn(T, T)
masked_scores = scores + mask
attn_weights = torch.softmax(masked_scores, dim=-1)
print(attn_weights[2, 3]) # tensor(0.)
# Production: torch.nn.functional.scaled_dot_product_attention
# with is_causal=True builds and applies this mask internally.Real products, models, and research that use this idea.
- Every decoder-only LLM from GPT-5.5 to Claude Opus 4.7 to Llama 4 Maverick uses this exact causal-mask pattern in self-attention.
- FlashAttention 3 generates the causal mask on the fly per tile to avoid materializing 32k x 32k matrices for long-context inference.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy use -inf rather than 0 as the mask sentinel for forbidden positions?
The mask is added to raw scores before softmax. Adding 0 leaves a score unchanged (allowing it through), adding -inf pushes the score to -inf, and exp(-inf) = 0 cleanly zeros the post-softmax weight. If you used 0 as the forbidden value, you would just be passing the raw score through and the model could still attend forward.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing row and column semantics. The row index is the query (the token doing the attending) and the column index is the key (the token being attended to). Row 2, column 3 means: token 2 looking at token 3, which is the forbidden direction.
60 second bullets to scan on the way to the call.
Row index vs column index meaning in an attention mask
Why the diagonal is allowed in a causal mask
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.