Where does FlashAttention deliver the biggest serving speedup?
FlashAttention's biggest win is long-context prefill, where avoiding HBM materialization of the n×n score matrix saves quadratic memory traffic.
Imagine grading every pair of students in a class by comparing their answers. The slow way writes every comparison into a giant grid on paper first, then adds up each row. With a big class that grid is enormous, and shuffling it on and off your desk is the real time sink. FlashAttention skips the giant grid. It walks through students in small groups, keeps a running tally on a sticky note, and never writes the full grid down. The answer is identical, but you barely touch the desk drawer. The bigger the class, the more drawer trips you save. A long prompt is a big class processed all at once, so that is exactly where skipping the grid helps most.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FlashAttention is the canonical example of an IO-aware algorithm: same math, dramatically less memory movement. Interviewers reach for it because it cleanly separates two kinds of candidate. One group thinks attention is bound by the matmul flops. The other group understands that on modern accelerators the binding constraint is moving data between high-bandwidth memory (HBM) and the on-chip SRAM where compute actually happens.
The question in front of you adds a second axis: not just what FlashAttention does, but where in a serving pipeline its benefit is largest. That distinction trips up people who memorized the headline but never reasoned about the workload. The answer hinges on one fact. The thing FlashAttention avoids materializing, the n×n score matrix, grows with the square of the sequence length, and the sequence is longest during prefill.
This deep dive builds the answer from the roofline up. We cover why attention is memory bound, what the naive kernel actually does to HBM, how tiling plus an online softmax removes the round-trips while keeping the result exact, and why that maps onto prefill rather than decode. By the end you should be able to defend the correct option and dismantle each distractor on memory-traffic grounds.
Why attention is memory-bound, not compute-bound
Every kernel on a GPU has an arithmetic intensity: flops performed per byte moved between HBM and the compute units. When intensity is low, the kernel spends its time waiting on memory, and the device's flop throughput is irrelevant. A modern accelerator like an H100 can issue tens of times more flops per second than it can stream bytes from HBM, so any kernel below that crossover ratio is memory-bound by definition.
Attention has low arithmetic intensity at the score stage. The matmuls themselves are not the problem; the n×n intermediate is. For a sequence of length n with head dimension d, the score matrix has n^2 entries, while the inputs and outputs are only on the order of n times d. So the kernel produces and consumes a large intermediate relative to the useful work it does on it. As n grows, the ratio of bytes moved to flops performed gets steadily worse, pushing the kernel deeper into the memory-bound regime.
The practical consequence is that wall-clock time is set by how many times that n×n matrix crosses the HBM boundary, not by how many multiply-adds the tensor cores can retire. This is the roofline insight that the whole technique rests on, and it is the framing an interviewer wants to hear first. If you can articulate why attention sits below the roofline knee, the rest of the FlashAttention story follows naturally. Candidates who skip this framing tend to misattribute the speedup to fewer flops, which is exactly the misconception the question is designed to expose.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Naive attention | FlashAttention |
|---|---|---|
| n×n score matrix in HBM | Materialized, read and written multiple times | Never materialized; tiled in SRAM |
| HBM traffic vs sequence length | Quadratic in n | Roughly linear in n |
| Numerical result | Exact | Exact, identical output |
| Biggest speedup regime | Baseline | Long-context prefill |
| Batch-1 decode benefit | Baseline | Small; no quadratic term to remove |
Real products, models, and research that use this idea.
- vLLM and SGLang use FlashAttention kernels for the prefill phase when serving Llama 4 and Qwen 3 on long prompts.
- TensorRT-LLM ships FlashAttention 3 on Hopper H100 and Blackwell B200 to accelerate long-context prefill in production.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is attention memory-bound rather than compute-bound in the first place?
Compare the arithmetic intensity. The score and value matmuls are modest flops, but the n×n intermediate forces large HBM reads and writes. On modern GPUs HBM bandwidth, not flops, is the scarce resource, so moving the matrix dominates wall-clock time.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming FlashAttention is an approximation or that it speeds up batch-1 decode most. It is exact, and its quadratic saving is largest during long-context prefill, not single-row decode.
60 second bullets to scan on the way to the call.
Why attention is memory bound rather than compute bound
What the n×n score matrix costs and why it scales quadratically
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.