Attention compares every token against every other token. For a sequence of length n, that produces an n×n grid of scores. A naive kernel writes that whole grid out to GPU main memory (HBM), reads it back to apply softmax, then reads it again to weight the values. The grid grows with the square of the sequence length, so moving it around dominates the cost.
**FlashAttention** is an IO-aware attention kernel. It computes the exact same result but never writes the full n×n grid to HBM. Instead it tiles the computation, keeps small blocks in fast on-chip SRAM, and maintains a running softmax so it can fuse the score, softmax, and value steps into one pass.
The payoff scales with how big that grid is. Prefill processes the whole prompt at once, so n is large and the saving is large. Batch-1 decode generates one token at a time, so each step only produces a single row of scores; there is almost no grid to avoid.
The correct answer is the long-context prefill option. FlashAttention is exact, not approximate, and its biggest win lands exactly where the sequence, and thus the score matrix, is longest.
**FlashAttention** is an IO-aware, exact attention algorithm. The bottleneck it attacks is memory traffic, not arithmetic. A naive kernel materializes the full n×n score matrix in HBM. It reads and writes that matrix several times for the softmax and value steps.
FlashAttention tiles the computation into blocks that fit in on-chip SRAM. It fuses the score, softmax, and value-weighting into a single pass using an online softmax. The full score matrix never touches HBM. So HBM traffic drops from quadratic to roughly linear in sequence length.
The saving scales with the size of that avoided matrix. Prefill ingests the entire prompt in parallel, so n is large and the quadratic term dominates. That is where FlashAttention delivers its biggest win. Reported numbers are roughly 5 to 10 times less attention HBM traffic, seen as 2 to 4 times faster end to end on long prompts.
Batch-1 decode is the opposite regime. Each step produces one query row, so the score matrix is effectively a vector. There is no quadratic cost to remove. Decode is bound by reading the model weights and the KV cache, not the score matrix. So FlashAttention helps little there. The result is identical to standard attention; it reorders memory access rather than approximating.
Quick reference
TL;DR
FlashAttention's biggest win is long-context prefill, where avoiding HBM materialization of the n×n score matrix saves quadratic memory traffic.
Imagine grading every pair of students in a class by comparing their answers. The slow way writes every comparison into a giant grid on paper first, then adds up each row. With a big class that grid is enormous, and shuffling it on and off your desk is the real time sink. FlashAttention skips the giant grid. It walks through students in small groups, keeps a running tally on a sticky note, and never writes the full grid down. The answer is identical, but you barely touch the desk drawer. The bigger the class, the more drawer trips you save. A long prompt is a big class processed all at once, so that is exactly where skipping the grid helps most.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Batch-1 autoregressive decode of short completions The per-step score matrix is one row, so there is no quadratic HBM cost for tiling to remove.
Very short sequences where the score matrix already fits comfortably in cache The quadratic term is tiny, so kernel launch and tiling overhead can outweigh the saved traffic.
Workloads bound by weight or KV cache streaming rather than attention scores Flash addresses score matrix traffic, so it cannot help a bottleneck that lives elsewhere.
Hardware or kernels without a supported tiled implementation Without an SRAM-tiled kernel for the target device, you fall back to materializing the matrix.
Quick scan · 2–4 min
Everything important, quickly.
3 min: attention is memory bound, naive kernel materializes the n×n matrix, tiling plus online softmax in SRAM, exact not approximate, prefill is the big win and decode is not.
Side-by-side
Aspect
Naive attention
FlashAttention
n×n score matrix in HBM
Materialized, read and written multiple times
Never materialized; tiled in SRAM
HBM traffic vs sequence length
Quadratic in n
Roughly linear in n
Numerical result
Exact
Exact, identical output
Biggest speedup regime
Baseline
Long-context prefill
Batch-1 decode benefit
Baseline
Small; no quadratic term to remove
Real-world examples
Real products, models, and research that use this idea.
1vLLM and SGLang use FlashAttention kernels for the prefill phase when serving Llama 4 and Qwen 3 on long prompts.
2TensorRT-LLM ships FlashAttention 3 on Hopper H100 and Blackwell B200 to accelerate long-context prefill in production.
3PyTorch exposes FlashAttention through scaled_dot_product_attention, so most 2026 training and inference stacks get it by default.
4DeepSeek V4 and Mistral Large 3 pipelines rely on FlashAttention to make long-sequence attention fit in HBM at all.
5Chunked-prefill serving in vLLM pairs FlashAttention with KV paging so long RAG prompts process without quadratic memory blowups.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is attention memory-bound rather than compute-bound in the first place?
Sign in or go Pro to see the approach for this follow-up.
QHow does the online softmax let FlashAttention avoid storing the full score row?
Sign in or go Pro to see the approach for this follow-up.
QWhy does batch-1 decode benefit so little while prefill benefits so much?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming FlashAttention is an approximation or that it speeds up batch-1 decode most. It is exact, and its quadratic saving is largest during long-context prefill, not single-row decode.