For a causal mask over a 4-token sequence, what value sits at row 2, column 3?
A causal attention mask is built for a sequence of length 4, using 0-indexed rows and columns. The convention is that allowed positions hold 0 and disallowed positions hold negative infinity, and the mask is added to the score matrix before softmax. Predict the value stored at row index 2 (the third query token), column index 3 (the fourth key token).
Position (2, 3) is strictly above the diagonal, so the causal mask holds -inf there and softmax zeros that attention weight after addition.
Imagine reading a book one word at a time and you are only allowed to look at words you have already read. Word number 3 (in zero-indexed counting, the third one) can look back at words 0, 1, and 2 (itself). It cannot peek ahead at word 4 because you have not read it yet. The causal mask is the rule that enforces 'no peeking ahead'. It works by writing a giant negative number on every forbidden position before the model decides where to look. After the model turns those numbers into voting weights, a giant negative number becomes essentially zero votes, so the forbidden word contributes nothing to what gets read next.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: row/column semantics, additive mask convention with 0 and -inf, why diagonal is allowed, softmax behavior on -inf, PyTorch torch.triu construction, on the fly generation in production kernels, prefill vs decode mask handling.
import torch
# Build the 4x4 causal mask used at attention.
# 0 = allowed, -inf = forbidden.
T = 4
mask = torch.triu(torch.full((T, T), float('-inf')), diagonal=1)
print(mask)
# tensor([[0., -inf, -inf, -inf],
# [0., 0., -inf, -inf],
# [0., 0., 0., -inf],
# [0., 0., 0., 0.]])
print(mask[2, 3]) # tensor(-inf)
# Applied to scores before softmax
scores = torch.randn(T, T)
masked_scores = scores + mask
attn_weights = torch.softmax(masked_scores, dim=-1)
print(attn_weights[2, 3]) # tensor(0.)
# Production: torch.nn.functional.scaled_dot_product_attention
# with is_causal=True builds and applies this mask internally.Real products, models, and research that use this idea.
- Every decoder-only LLM from GPT-5.5 to Claude Opus 4.7 to Llama 4 Maverick uses this exact causal-mask pattern in self-attention.
- FlashAttention 3 generates the causal mask on the fly per tile to avoid materializing 32k x 32k matrices for long-context inference.
- PyTorch's torch.nn.functional.scaled_dot_product_attention exposes the causal flag, which internally builds this same upper-triangular -inf mask.
- vLLM and TensorRT-LLM use causal masking at prefill but skip the mask entirely at decode since the new token sees all prior cached positions by construction.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy use -inf rather than 0 as the mask sentinel for forbidden positions?
QHow does the causal mask interact with KV-cache decode?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Confusing row and column semantics. The row index is the query (the token doing the attending) and the column index is the key (the token being attended to). Row 2, column 3 means: token 2 looking at token 3, which is the forbidden direction.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.