A causal attention mask is built for a sequence of length 4, using 0-indexed rows and columns. The convention is that allowed positions hold 0 and disallowed positions hold negative infinity, and the mask is added to the score matrix before softmax. Predict the value stored at row index 2 (the third query token), column index 3 (the fourth key token).
**The answer is -inf.**
A causal mask enforces the rule that an autoregressive token may only attend to positions at or before itself. In a 4-token sequence, position 2 can attend to positions 0, 1, and 2, but not to position 3 (which comes later).
The convention used here is the additive mask: allowed positions hold 0, disallowed positions hold -inf. The mask is added element-wise to the raw attention scores before softmax. Adding 0 leaves a score unchanged; adding -inf pushes it to the most negative number representable, and softmax sends `exp(-inf) = 0` so the weight becomes exactly zero.
Visually for a 4x4 mask:
```
[0, -inf, -inf, -inf]
[0, 0, -inf, -inf]
[0, 0, 0, -inf] <- row 2
[0, 0, 0, 0 ]
```
Position (2, 3) sits strictly above the diagonal in the row-2 entry, and holds -inf.
In PyTorch this matrix is built with `torch.triu(torch.full((4,4), float('-inf')), diagonal=1)`, which puts -inf strictly above the diagonal. The `diagonal=1` argument is what makes the diagonal itself allowed (so a token can always attend to itself).
**Answer: -inf** (additive mask convention).
A causal mask says: a query at position `i` may only attend to keys at positions `j <= i`. Position (2, 3) means query token 2 trying to attend to key token 3, which would be attending forward in time and is forbidden in an autoregressive model.
The additive mask M is constructed so that:
$$M_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}$$
It is added to the raw score matrix before softmax:
$$\text{attn} = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)$$
At entry (2, 3), we have `i = 2`, `j = 3`, so `i < j` and the mask is -inf. After softmax that entry's weight is `exp(-inf) / (...) = 0`, so token 2 gives token 3 exactly zero attention weight.
In PyTorch: `torch.triu(torch.full((T, T), float('-inf')), diagonal=1)`. The `diagonal=1` shift is what keeps the diagonal itself allowed (a token can attend to itself).
Quick reference
TL;DR
Position (2, 3) is strictly above the diagonal, so the causal mask holds -inf there and softmax zeros that attention weight after addition.
Imagine reading a book one word at a time and you are only allowed to look at words you have already read. Word number 3 (in zero-indexed counting, the third one) can look back at words 0, 1, and 2 (itself). It cannot peek ahead at word 4 because you have not read it yet. The causal mask is the rule that enforces 'no peeking ahead'. It works by writing a giant negative number on every forbidden position before the model decides where to look. After the model turns those numbers into voting weights, a giant negative number becomes essentially zero votes, so the forbidden word contributes nothing to what gets read next.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Encoder-only or bidirectional models like BERT and RoBERTa Encoders attend in both directions to build full-context representations, no causal restriction applies and the mask matrix is symmetric (or absent).
Cross-attention from a decoder to an encoder (T5, BART) The encoder is fully observed at every decoder step, the cross-attention layer does not apply a causal mask on the encoder axis.
Prefix-LM training (UL2-style, T5-1.1) where the prefix is bidirectional The prefix portion of the sequence is unmasked (full attention), only the suffix has the causal restriction. The mask is hybrid, not pure upper-triangular.
Quick scan · 2–4 min
Everything important, quickly.
4 min: row/column semantics, additive mask convention with 0 and -inf, why diagonal is allowed, softmax behavior on -inf, PyTorch torch.triu construction, on the fly generation in production kernels, prefill vs decode mask handling.
Code example
python
import torch
# Build the 4x4 causal mask used at attention.
# 0 = allowed, -inf = forbidden.
T = 4
mask = torch.triu(torch.full((T, T), float('-inf')), diagonal=1)
print(mask)
# tensor([[0., -inf, -inf, -inf],
# [0., 0., -inf, -inf],
# [0., 0., 0., -inf],
# [0., 0., 0., 0.]])
print(mask[2, 3]) # tensor(-inf)
# Applied to scores before softmax
scores = torch.randn(T, T)
masked_scores = scores + mask
attn_weights = torch.softmax(masked_scores, dim=-1)
print(attn_weights[2, 3]) # tensor(0.)
# Production: torch.nn.functional.scaled_dot_product_attention
# with is_causal=True builds and applies this mask internally.
Real-world examples
Real products, models, and research that use this idea.
1Every decoder-only LLM from GPT-5.5 to Claude Opus 4.7 to Llama 4 Maverick uses this exact causal-mask pattern in self-attention.
2FlashAttention 3 generates the causal mask on the fly per tile to avoid materializing 32k x 32k matrices for long-context inference.
3PyTorch's torch.nn.functional.scaled_dot_product_attention exposes the causal flag, which internally builds this same upper-triangular -inf mask.
4vLLM and TensorRT-LLM use causal masking at prefill but skip the mask entirely at decode since the new token sees all prior cached positions by construction.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy use -inf rather than 0 as the mask sentinel for forbidden positions?
Sign in or go Pro to see the approach for this follow-up.
QHow does the causal mask interact with KV-cache decode?
Sign in or go Pro to see the approach for this follow-up.
QWhy do modern kernels generate the mask on the fly rather than materializing it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing row and column semantics. The row index is the query (the token doing the attending) and the column index is the key (the token being attended to). Row 2, column 3 means: token 2 looking at token 3, which is the forbidden direction.