Why is causal masking non-negotiable in autoregressive pretraining?
The causal mask stops each position from attending to later tokens, so every next-token prediction in training uses only the past, exactly as in generation.
Picture a student practicing for a quiz where each question is: what is the next word? The practice sheet shows a whole sentence, so before each guess the student slides a card over every word to the right. If the card slips, the next word is in plain view. The student scores perfectly and learns nothing, because on quiz day the next word is not printed yet. The causal mask is that card. The model practices on a full sentence at once, with every position guessing its own next word. The mask makes sure no position can see the words after it. Practice then matches the real quiz, where the model writes one word at a time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A decoder-only language model is trained to predict each next token from the tokens before it. Training processes a whole sequence in parallel, so something must stop each position from reading the tokens after it. That something is the causal mask.
The mask is a small piece of code, but it decides whether the training objective is honest. Get it right and parallel training, left-to-right generation, and the KV cache all agree. Get it wrong and the model learns to copy, the loss looks excellent, and generation fails.
What plain attention would see
Pretraining a decoder-only model minimizes the next-token loss at every position of a sequence at the same time:
The whole sequence enters the model in one forward pass. The input at position t is token x_t, and the target is token x_{t+1}: the same sequence shifted left by one. This is teacher forcing. The model always conditions on the true previous tokens, not on its own guesses.
Self-attention by itself has no sense of direction. Every query can score every key. So without extra help, the hidden state at position t is built partly from position t+1, which holds the target. Gradient descent finds the shortest path to low loss, and here that path is copying. The model learns to read the answer from the next slot instead of learning language.
The damage shows up only at generation time. When the model writes text, position t+1 is empty, because that token is what the model must produce. A network that learned to copy from a slot that is now empty has nothing useful to say.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- PyTorch's torch.nn.functional.scaled_dot_product_attention takes is_causal=True and applies the causal mask inside the fused attention kernel.
- FlashAttention exposes a causal=True flag and skips the blocks above the diagonal, so causal attention needs about half the score computation.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can training still be parallel if masking is causal?
Teacher forcing feeds the true previous tokens, so all positions are known up front; the mask only limits which of them each position may read.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the causal mask as a speed or memory trick. It is a rule about information: training still runs all positions in one parallel pass, and the mask only hides the future tokens.
60 second bullets to scan on the way to the call.
Mask values above, on, and below the diagonal
Why minus infinity before the softmax gives exactly zero weight
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.