FlashAttention
Also known as: FlashAttention-2, FlashAttention-3
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
An IO-aware exact attention algorithm that avoids materializing the full N×N attention matrix in HBM by tiling the computation and keeping intermediates in fast SRAM. 2-4x faster training and inference with no quality change.
In practice
The reason long-context training stopped being prohibitively slow. Senior infra interviews probe the tiling strategy and HBM vs SRAM bandwidth.
Related topics
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.