Why is a causal mask required for autoregressive language model training?

Decoder LLMs are trained with **teacher forcing**: the whole ground truth sequence is fed in at once, and every position predicts the next token in parallel. Position 1 predicts token 2, position 2 predicts token 3, and so on: all in a single forward pass. That's what makes training fast. The problem: a normal self-attention layer at position t can look at every position in the input, including position t+1, which holds the very token the model is supposed to predict. Without intervention, the model trivially learns 'just attend to t+1 and copy the value', a leak that makes loss collapse but makes the model useless at inference, where t+1 doesn't exist yet. The **causal mask** is the fix. Before applying softmax to attention scores, it adds `-∞` to scores at all positions `j > i`. After softmax, those scores become zero weight. Position t can attend to positions 1..t, never to t+1.

The trap here is thinking the mask exists for inference, it doesn't. Decoder LLMs train with teacher forcing, the whole target sequence is fed in once and every position predicts the next token in parallel. That's what gives you efficient training. But if you don't mask, position t can attend to position t+1, which literally holds the target it's supposed to predict. The model would learn to attend forward and copy, loss would collapse, and at inference the trick wouldn't work because the future tokens don't exist yet. So you add a **causal mask**, a lower triangular pattern that adds `-∞` to attention scores for any position `j > i` before softmax: $$M_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}$$ After softmax, those future positions get exactly zero weight. The mask is needed at TRAINING: that's the key point most people miss. At inference, generation is naturally causal because you only have the tokens generated so far; there's no `t+1` to attend to. One tradeoff worth mentioning: the mask doesn't actually reduce FLOPs in a vanilla PyTorch implementation. You still compute the full `T × T` attention matrix; you just zero out half. FlashAttention v2 and v3 can exploit the triangular structure to skip blocks entirely and get roughly 2x speedup on causal attention.