Causal Mask
Also known as: Autoregressive mask, Triangular mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
An attention mask that prevents each position from attending to future positions, enforcing autoregressive generation. Implemented by adding -∞ to disallowed positions in the attention scores before softmax.
In practice
Why decoder-only models can be trained on full sequences in parallel. Expect questions on its memory/compute implications.
Related topics
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.