FlashAttention computes exactly the same attention output as a naive implementation: same logits, same softmax, same V-weighted sum. So where does the 2-4× speedup come from on serving workloads? Where is the win biggest and where is it smallest?
Free · 2 AI evals / day
**FlashAttention** computes exactly the same attention output as a textbook implementation. Same logits, same softmax, same weighted sum of values. So the speedup cannot come from doing less arithmetic.
The win comes from how the GPU moves data. A GPU has a tiny pool of very fast on-chip memory called SRAM and a large pool of slower off-chip memory called HBM. The naive attention kernel writes the full attention score matrix to HBM, reads it back to run softmax, then reads it again to weight the values. That score matrix is n by n, so at long context it is enormous, and shuttling it to and from HBM dominates the time.
FlashAttention is IO-aware. It splits the queries, keys, and values into small tiles that fit in SRAM, and computes the scores, softmax, and value-weighted sum for each tile without ever writing the full matrix out. A running-max and running-denominator trick lets it finish softmax correctly tile by tile.
The result is far less HBM traffic, which is the real bottleneck. The math is untouched, so the output is identical down to floating point rounding. The benefit is largest where the score matrix is biggest: long prompts during prefill.
**FlashAttention** changes nothing about the math. Same logits, same softmax, same value-weighted sum. The output is exact, not an approximation.
What changes is the memory access pattern. A GPU reads HBM far slower than its on-chip SRAM. Naive attention writes the full n-by-n score matrix to HBM, reads it back for softmax, then reads it again to weight values. Attention is memory bound, so those round-trips dominate.
FlashAttention is IO-aware. It tiles Q, K, and V into SRAM-resident blocks. It fuses scores, softmax, and the value sum into one kernel. The score matrix is never written to HBM. The online-softmax trick keeps a running max and denominator. So each tile updates the result correctly without seeing all scores at once. HBM traffic drops from order n-squared to order n times d.
The win is biggest in prefill on long context, where the matrix is huge. It shows up as roughly 2 to 4 times end to end. The win is smallest in batch-1 decode. There each step does one attention row, so the dominant HBM cost is streaming the model weights, not the score matrix.
Quick reference
TL;DR
FlashAttention is IO-aware: tiling plus online softmax keep attention in SRAM, so HBM traffic drops from O(n^2) to O(n*d). Same math, fewer memory round-trips, biggest win in prefill.
Imagine doing a huge multiplication by hand. The naive way is to write every intermediate number onto a giant sheet of paper on the far side of the room, walk over to read it back, then walk over again to write the next step. The paper is slow to reach, so most of your time goes to walking, not multiplying. FlashAttention keeps a small notepad in your pocket. It works on one small chunk at a time on the notepad, keeps a running total, and never writes the giant sheet at all. The final answer is identical, because the arithmetic is the same. You just stopped wasting time walking back and forth to the far paper, so the whole job finishes much faster.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-stream batch-1 decode with short context There is no large score matrix to avoid; the step is bound by reading weights from HBM.
Very short sequences where attention is not the bottleneck The n-squared term is tiny, so kernel launch and other layers dominate the runtime.
Hardware or frameworks without a tuned FlashAttention kernel for the dtype An untuned fused kernel can underperform a well-optimized library matmul path.
When you actually need the full score matrix as an output FlashAttention never materializes it, so attention-weight inspection requires a different kernel.
Quick scan · 2–4 min
Everything important, quickly.
4 min: exact math unchanged + SRAM versus HBM + naive kernel materializes n-by-n in HBM + tiling and fusion + online softmax + traffic drops to O(n*d) + prefill wins, batch-1 decode does not.
Side-by-side
Aspect
Naive attention
FlashAttention
Output
Exact
Exact, identical
Score matrix in HBM
Materialized, n by n
Never written
HBM traffic
Order n-squared
Order n times d
Peak memory
Order n-squared
Order n
Biggest win
n/a
Long-context prefill
Real-world examples
Real products, models, and research that use this idea.
1vLLM and SGLang both ship FlashAttention kernels as the default prefill attention backend for serving Llama 4 and Qwen 3 in 2026.
2NVIDIA TensorRT-LLM uses fused FlashAttention-style kernels for long-context prefill on H100 and B200, where HBM bandwidth is the binding constraint.
3PyTorch exposes the same fused exact attention through scaled_dot_product_attention, which selects a FlashAttention backend automatically when shapes allow.
4Frontier long-context models like Gemini 3.1 Pro and Claude Opus 4.7 rely on IO-aware attention to make 128k-plus prompt prefill economically feasible.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the online-softmax recurrence stay numerically exact across tiles?
Sign in or go Pro to see the approach for this follow-up.
QWhy is decode at batch 1 memory bound on weights rather than on the score matrix?
Sign in or go Pro to see the approach for this follow-up.
QHow does FlashAttention-2 improve on the original kernel?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming FlashAttention changes the math or approximates softmax. It is exact; the only thing that changes is the memory access pattern, not a single output number.