FlashAttention is an exact algorithm. Same output as naive attention up to floating-point reduction order; only the HBM and SRAM traffic pattern changes.
Imagine you have to add up a million numbers and your scratchpad only fits a few hundred at a time. The slow way is to keep running back to a giant whiteboard, copying chunks in and partial sums out, until you finish. The fast way is to load one batch into your scratchpad, finish all the math you can on it, then load the next batch, and combine results in your head as you go. You get the exact same total either way, what changes is how much you walked to the whiteboard. FlashAttention is the fast way for attention: same answer, far less walking.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FlashAttention is one of the cleanest examples of an I/O-aware algorithm in modern ML systems. The output is bit for bit equivalent to naive attention (modulo floating-point reduction order), but the way bytes move between HBM and on-chip SRAM is completely different.
The distinction matters because the field has many fast-attention variants and they fall into two very different categories. Some, Performer, Linformer, sparse Transformer, change the math itself and trade exactness for speed. FlashAttention changes only how the exact math is scheduled. This deep dive walks the standard attention I/O pattern, why HBM bandwidth (not FLOPs) bottlenecks long-sequence attention, the online softmax trick that makes tiling exact, the backward-pass details that surprise people, and the broader landscape of fast-attention techniques.
Standard attention I/O is the bottleneck
The textbook implementation of attention is short to write and slow to run on long sequences. The math:
What the GPU actually does
- Compute
S = QK^T / sqrt(d_k). An n-by-n matrix, written to HBM. - Read
Sback, compute row-wise softmax, writeP = softmax(S)to HBM. Another n-by-n matrix. - Read
Pback, computeO = P @ V. WriteOto HBM.
Four passes over an n-by-n object. At n = 8192 with fp16, that is roughly 128 MB per attention head per layer, just for the intermediate matrices. For a 32-layer 32-head model that is gigabytes of HBM traffic per forward pass.
Why the FLOPs are not the issue
The FLOPs in attention are roughly 4 * n^2 * d_model per layer. On an H100, the tensor cores can do this in tens of microseconds. The actual wall-clock time at long n is dominated by waiting for HBM reads and writes of the intermediate matrices. The kernel is bandwidth-bound, and the bandwidth is being spent moving an object (the attention matrix) that nobody needs to keep.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- PyTorch torch.nn.functional.scaled_dot_product_attention dispatches to FlashAttention v2 on supported GPUs as of PyTorch 2.x.
- Llama 4 Maverick training and inference pipelines use FlashAttention v3 as the default attention kernel on Hopper hardware.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf FlashAttention is exact, why do training runs sometimes show small numerical differences between FlashAttention and a naive PyTorch attention?
Floating-point addition is non-associative. FlashAttention sums tiles in a different order than the naive implementation, so the bitwise output can shift by a few ULPs. Loss curves match in expectation but seeds may not be bitwise reproducible across kernel choices unless you pin the implementation.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling FlashAttention an approximation. It is not. Performer and Linformer approximate; FlashAttention reorganizes the I/O of the exact same softmax(QK^T/sqrt(d_k))V.
60 second bullets to scan on the way to the call.
Definition of exact algorithm versus approximate algorithm
Standard attention formula and which step writes to HBM
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.