Where does FlashAttention deliver the biggest serving speedup?
FlashAttention's biggest win is long-context prefill, where avoiding HBM materialization of the n×n score matrix saves quadratic memory traffic.
Imagine grading every pair of students in a class by comparing their answers. The slow way writes every comparison into a giant grid on paper first, then adds up each row. With a big class that grid is enormous, and moving it on and off your desk takes most of the time. FlashAttention skips the giant grid. It walks through students in small groups, keeps a running tally on a sticky note, and never writes the full grid down. The answer is identical, but you barely touch the desk drawer. The bigger the class, the more drawer trips you save. A long prompt is a big class processed all at once, so that is exactly where skipping the grid helps most.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FlashAttention is an IO-aware attention algorithm: it does the same math as standard attention with much less memory movement. For standard attention on GPUs, the limit is not the matmul FLOPs. The limit is moving data between high-bandwidth memory (HBM) and the on-chip SRAM where compute happens.
Where in a serving pipeline the benefit is largest depends on one fact. The object FlashAttention avoids writing to HBM, the n×n score matrix, grows with the square of the sequence length, and the sequence is longest during prefill.
Why attention is memory-bound, not compute-bound
Every kernel on a GPU has an arithmetic intensity: flops performed per byte moved between HBM and the compute units. When intensity is low, the kernel spends its time waiting on memory, and the device's flop throughput does not set the speed. An H100 SXM does about 989 dense BF16 TFLOPS against 3.35 TB/s of HBM bandwidth, about 295 FLOPs per byte. Any kernel below that ratio is memory-bound.
Attention has low arithmetic intensity at the score stage. The matmuls themselves are not the problem; the n×n intermediate is. For a sequence of length n with head dimension d, the score matrix has n^2 entries, while the inputs and outputs are only on the order of n times d. So the kernel produces and consumes a large intermediate relative to the useful work it does on it. The softmax, masking and dropout steps do only a few FLOPs per element they read and write. As n grows, that traffic grows with n^2, and the kernel stays memory-bound.
In practice, wall-clock time depends on how many times that n×n matrix crosses the HBM boundary. It does not depend on how many multiply-adds the tensor cores can complete. FlashAttention rests on this roofline fact. Its speedup comes from fewer HBM reads and writes, not from fewer FLOPs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Naive attention | FlashAttention |
|---|---|---|
| n×n score matrix in HBM | Materialized, read and written multiple times | Never materialized; tiled in SRAM |
| HBM traffic vs sequence length | Quadratic in n | Many times lower (about n^2 x d^2 / M accesses, M = SRAM size) |
| Numerical result | Exact | Exact, identical output |
| Biggest speedup regime | Baseline | Long-context prefill |
| Batch-1 decode benefit | Baseline | Small; no quadratic term to remove |
Real products, models, and research that use this idea.
- vLLM and SGLang use FlashAttention kernels for the prefill phase when serving Llama 4 and Qwen3 on long prompts.
- TensorRT LLM runs the context (prefill) phase with a single fused attention kernel, which uses the FlashAttention algorithm for longer sequences.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is attention memory-bound rather than compute-bound in the first place?
Compare the arithmetic intensity. The score and value matmuls are modest flops, but the n×n intermediate forces large HBM reads and writes. For this kernel, HBM bandwidth, not flops, is the limit, so moving the matrix dominates wall-clock time.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming FlashAttention is an approximation or that it speeds up batch-1 decode most. It is exact, and its quadratic saving is largest during long-context prefill, not single-row decode.
60 second bullets to scan on the way to the call.
Why attention is memory bound rather than compute bound
What the n×n score matrix costs and why it scales quadratically
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.