Why does FlashAttention speed up serving even though it doesn't change the math?
FlashAttention computes exactly the same attention output as a naive implementation: same logits, same softmax, same V weighted sum. So where does the 2-4× speedup come from on serving workloads? Where is the win biggest and where is it smallest?
FlashAttention is IO-aware: tiling plus online softmax keep attention in SRAM, so HBM traffic drops from O(n^2) to O(n*d). Same math, fewer memory round-trips, biggest win in prefill.
Imagine doing a huge multiplication by hand. The naive way is to write every intermediate number onto a giant sheet of paper on the far side of the room, walk over to read it back, then walk over again to write the next step. The paper is slow to reach, so most of your time goes to walking, not multiplying. FlashAttention keeps a small notepad in your pocket. It works on one small chunk at a time on the notepad, keeps a running total, and never writes the giant sheet at all. The final answer is identical, because the arithmetic is the same. You just stopped wasting time walking back and forth to the far paper, so the whole job finishes much faster.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FlashAttention is the canonical example of an optimization that makes a GPU kernel faster without changing a single output number. That property is exactly why it is a sharp interview question. If the math is identical, the speedup must come from somewhere other than arithmetic, and a candidate who cannot say where has not understood how modern accelerators actually spend their time.
The answer lives in the memory hierarchy. A modern GPU pairs an enormous matrix-multiply throughput with a comparatively narrow memory pipe to its large off-chip memory. Many transformer kernels, attention chief among them, do too little arithmetic per byte loaded to keep the compute units busy. They stall waiting on memory. The naive attention implementation makes this far worse by writing the entire score matrix out to off-chip memory and reading it back twice.
This deep dive walks through why attention is memory bound, what the naive kernel wastes, how tiling and the online-softmax recurrence let FlashAttention avoid materializing the score matrix while staying mathematically exact, and why the payoff is concentrated in long-context prefill rather than single-stream decode. By the end you should be able to reason about the roofline, do the traffic math, and explain the prefill-versus-decode asymmetry on a whiteboard.
Why attention is memory bound, not compute bound
Performance on a GPU is governed by the roofline model. A kernel is compute bound when it does many arithmetic operations per byte it loads, and memory bound when it does few. The dividing line is the hardware's ratio of peak FLOPs to peak memory bandwidth.
The two memory tiers that matter are SRAM and HBM. SRAM is the small, extremely fast on-chip scratchpad, measured in tens of kilobytes per streaming multiprocessor. HBM is the large off-chip memory, tens to hundreds of gigabytes, but with bandwidth roughly an order of magnitude lower than SRAM. Anything you read or write in HBM is comparatively slow.
Attention has low arithmetic intensity. For each element of the score matrix you do a dot product, but the softmax and value-weighting steps move a lot of data per FLOP. So the kernel spends most of its time waiting on HBM, not on the math units. That is the crucial framing: attention is bound by bytes moved, not by floating point operations performed.
This matters because it tells you which optimizations can help and which cannot. Throwing more raw compute at a memory-bound kernel does nothing, since the math units already sit idle. The only lever that helps is moving fewer bytes over the slow link, or keeping more of the work resident in the fast tier. A candidate who reaches for matmul tricks or lower-precision multiplies to explain the FlashAttention win has misdiagnosed the bottleneck. The right instinct is to ask where the data lives and how often it crosses the HBM boundary.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Naive attention | FlashAttention |
|---|---|---|
| Output | Exact | Exact, identical |
| Score matrix in HBM | Materialized, n by n | Never written |
| HBM traffic | Order n-squared | Order n times d |
| Peak memory | Order n-squared | Order n |
| Biggest win | n/a | Long-context prefill |
Real products, models, and research that use this idea.
- vLLM and SGLang both ship FlashAttention kernels as the default prefill attention backend for serving Llama 4 and Qwen 3 in 2026.
- NVIDIA TensorRT-LLM uses fused FlashAttention-style kernels for long-context prefill on H100 and B200, where HBM bandwidth is the binding constraint.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the online-softmax recurrence stay numerically exact across tiles?
Track a running max and a running sum of exponentials. When a new block arrives with a larger max, rescale the previous accumulator and denominator by the exponential of the old max minus the new max before adding the block's contribution.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming FlashAttention changes the math or approximates softmax. It is exact; the only thing that changes is the memory access pattern, not a single output number.
60 second bullets to scan on the way to the call.
Why an identical output still runs faster on a GPU
The SRAM versus HBM memory hierarchy and why attention is memory bound
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.