FlashAttention
Also known as: FlashAttention-2, FlashAttention-3
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
An IO-aware exact attention algorithm that avoids materializing the full N×N attention matrix in HBM by tiling the computation and keeping intermediates in fast SRAM. 2-4x faster training and inference with no quality change.
In practice
The reason long-context training stopped being prohibitively slow. Senior infra interviews probe the tiling strategy and HBM vs SRAM bandwidth.
Related topics
Questions that mention this term
- Match each long context strategy to what it modifies.
- Spot the masking bug in this packed sequence training setup
- Walk through what FlashAttention does differently from standard attention, and what changes between v1, v2, and v3.
- Does FlashAttention change the attention output relative to standard attention?
- Why is causal attention cheaper than full bidirectional at the same length?
- Flash-Decoding parallelizes an axis FlashAttention v2 left alone. Pick which one.
Related terms
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.
Greedy Decoding
At each step, pick the single highest-probability token. Fast and deterministic, but often loops.