What does FlashAttention v1 actually change vs standard attention?
FlashAttention tiles Q, K, V into SRAM and uses online softmax to skip the n×n write to HBM. Same math, 2-4x faster.
Imagine adding up scores between every page of a huge book and every other page. The slow way spreads all pages on a giant table and runs back and forth from the shelf. Picture a smaller desk that only fits a few pages at a time. You bring a stack over, score them against a passing stack, jot a running total, and swap stacks. The full grid of scores never sits anywhere; the running total carries the answer. The desk is small but fast to reach, so the work flies. The shelf trips were the slow part, and you cut them out.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Explain why standard attention is memory bound, walk the tiling and online softmax, contrast forward (tile + save m,l) vs backward (recompute), then trace v1 → v2 (seq dim parallelism) → v3 (Hopper async, FP8) and emphasize the math equivalence throughout.
# Online softmax (the heart of FlashAttention's tiling correctness)
# Process a row in chunks; maintain running max m and denominator l
# so the final result matches softmax over the full row.
import math
def online_softmax_chunked(scores_chunks):
m = -math.inf # running max
l = 0.0 # running denominator (sum of exp(x - m))
out_numer = 0.0 # running numerator if multiplied by values
for chunk in scores_chunks:
m_new = max(m, max(chunk))
# Rescale previous accumulators to new max
l = l * math.exp(m - m_new) + sum(math.exp(s - m_new) for s in chunk)
out_numer = out_numer * math.exp(m - m_new) # would multiply by chunk values too
m = m_new
# Final softmax denominator is l; one division finalizes the row.
return l, m| Version | Year | Key change | Typical speedup |
|---|---|---|---|
| FlashAttention v1 | 2022 | Tiling + online softmax; O(n) memory; same math | 2-4x vs standard |
| FlashAttention v2 | 2023 | Sequence dim parallelism, fewer non-matmul FLOPs | ~2x vs v1 |
| FlashAttention v3 | 2024 | Hopper async (TMA, warpgroup MMA), FP8 path | ~1.5-2x vs v2 on H100 |
Real products, models, and research that use this idea.
- PyTorch 2 and later ship F.scaled_dot_product_attention, which dispatches to FlashAttention-2/3 kernels on supported GPUs.
- vLLM, TensorRT-LLM, and SGLang all use FlashAttention style kernels in production serving for Llama 4 Maverick, DeepSeek V4, and Qwen 3.5.
- Training 8B-class models at 128k context on H100/H200 nodes in 2026 is feasible only with FlashAttention memory; naive attention would OOM at a fraction of that length.
- FlashAttention-3 with FP8 is the default attention path on Hopper for many May 2026 serving stacks, including pipelines that back Claude Opus 4.7 long context endpoints.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the backward pass of FlashAttention avoid storing the n×n attention matrix?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Calling FlashAttention an 'approximation', it is mathematically equivalent to standard attention; only the I/O pattern changes.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.