Causal Mask
Also known as: Autoregressive mask, Triangular mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
An attention mask that prevents each position from attending to future positions, enforcing autoregressive generation. Implemented by adding -∞ to disallowed positions in the attention scores before softmax.
In practice
Why decoder-only models can be trained on full sequences in parallel. Expect questions on its memory/compute implications.
Related topics
Questions that mention this term
- Spot the masking bug in this packed sequence training setup
- Spot the error: 'I…
- Predict the softmax output for a causal masked attention row.
- DeepSeek trains models to predict K tokens per step. Pick how that stays causal.
- Why is causal attention cheaper than full bidirectional at the same length?
- At which step in the attention pipeline does the mask actually take effect?
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Context Window
The max number of tokens a model can attend to at once.
Decoder-Only
Single autoregressive transformer stack: the shape of every modern frontier LLM.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
Grouped-Query Attention (GQA)
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.