What does FlashAttention v1 actually change vs standard attention?
FlashAttention tiles Q, K, V into SRAM and uses online softmax to skip the n×n write to HBM. Same math, 2-4x faster.
Imagine adding up scores between every page of a huge book and every other page. The slow way spreads all pages on a giant table and runs back and forth from the shelf. Picture a smaller desk that only fits a few pages at a time. You bring a stack over, score them against a passing stack, jot a running total, and swap stacks. The full grid of scores never sits anywhere; the running total carries the answer. The desk is small but fast to reach, so the work flies. The shelf trips were the slow part, and you cut them out.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FlashAttention is the kernel work that, more than any single architectural change, made long context LLM training and serving feasible. Most engineers learn it as a checkbox optimization, flip on F.scaled_dot_product_attention and move on. The underlying lesson is structural: on modern GPUs, attention is memory bandwidth bound rather than compute bound, and that single fact reshapes the whole design.
This deep dive walks why standard attention is bandwidth bound, what v1's tiling and online softmax actually compute, what v2 reorganized in the loop structure, what v3 inherits from Hopper-specific hardware, the exact vs approximate point that gets misstated constantly, and what the production picture looks like in 2026. The goal is to give you a model of FlashAttention that survives in conversation with a kernel engineer, not just a paper citation.
The HBM/SRAM bottleneck: why standard attention is bandwidth bound
An A100 sustains roughly 312 TFLOPs of FP16 tensor core throughput against about 2 TB/s of HBM bandwidth. That ratio works out to around 150 FLOPs per byte of HBM traffic: the chip needs to do 150 multiplies for every byte it reads from HBM to stay saturated. A kernel that reads or writes more bytes than it has compute to spend on them stalls the tensor cores.
Standard attention falls into that trap structurally. The n×n score matrix is written after QK^T, read back for the softmax (which needs the full row to compute the denominator), written after the softmax, and read once more for the V multiply. Every one of those passes moves O(n²) bytes across the slow HBM bus. At n = 4096 with 32 heads and FP16 that's 16 million entries × 2 bytes × 32 heads = 1 GB per layer per pass, multiplied by 4 passes.
The matmuls themselves are not the problem. They finish quickly on the tensor cores and then sit idle waiting on the next load. Notice what this implies for naive optimizations: swapping in a faster matmul kernel does nothing here, because the bottleneck is bytes per second on the HBM bus rather than FLOPs-per-second on the tensor cores. The right move is to keep the score matrix off HBM entirely. That's the insight FlashAttention monetizes.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Online softmax (the heart of FlashAttention's tiling correctness)
# Process a row in chunks; maintain running max m and denominator l
# so the final result matches softmax over the full row.
import math
def online_softmax_chunked(scores_chunks):
m = -math.inf # running max
l = 0.0 # running denominator (sum of exp(x - m))
out_numer = 0.0 # running numerator if multiplied by values
for chunk in scores_chunks:
m_new = max(m, max(chunk))
# Rescale previous accumulators to new max
l = l * math.exp(m - m_new) + sum(math.exp(s - m_new) for s in chunk)
out_numer = out_numer * math.exp(m - m_new) # would multiply by chunk values too
m = m_new
# Final softmax denominator is l; one division finalizes the row.
return l, m| Version | Year | Key change | Typical speedup |
|---|---|---|---|
| FlashAttention v1 | 2022 | Tiling + online softmax; O(n) memory; same math | 2-4x vs standard |
| FlashAttention v2 | 2023 | Sequence dim parallelism, fewer non-matmul FLOPs | ~2x vs v1 |
| FlashAttention v3 | 2024 | Hopper async (TMA, warpgroup MMA), FP8 path | ~1.5-2x vs v2 on H100 |
Real products, models, and research that use this idea.
- PyTorch 2 and later ship F.scaled_dot_product_attention, which dispatches to FlashAttention-2/3 kernels on supported GPUs.
- vLLM, TensorRT-LLM, and SGLang all use FlashAttention style kernels in production serving for Llama 4 Maverick, DeepSeek V4, and Qwen 3.5.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the backward pass of FlashAttention avoid storing the n×n attention matrix?
Forward saves only Q, K, V plus the per-row m and l. Backward RECOMPUTES the attention matrix on the fly from those, tiled the same way. Trades a small FLOP overhead for huge memory savings, favorable on memory bound attention.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling FlashAttention an 'approximation', it is mathematically equivalent to standard attention; only the I/O pattern changes.
60 second bullets to scan on the way to the call.
Why standard attention is bound by memory bandwidth on modern GPUs
The tiling pattern: Q, K, V blocks staged through SRAM
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.