**FlashAttention** is an exact algorithm. It produces the same output as the textbook attention formula, the difference is purely in how the computation is scheduled on a GPU.
The naive implementation builds the full attention matrix `S = QK^T` in HBM (high-bandwidth memory), runs softmax over rows, multiplies by V, and writes the result back. For sequence length n that matrix has `n * n` entries, which is the dominant memory cost and also the main source of HBM bandwidth pressure.
FlashAttention reorganizes the work so that the n-by-n matrix never lives in HBM. It tiles Q, K, and V into blocks small enough to fit in the GPU's on-chip SRAM (a few hundred KB per SM), runs attention block by block using an online softmax that updates a running maximum and a running denominator as new blocks arrive, and writes only the final output back to HBM.
The key word is **exact**. Not 'good enough', not 'approximately the same', the same answer up to the floating-point order of summation. That is why FlashAttention shipped as a drop-in replacement in PyTorch with no model-quality regression.
**FlashAttention is an exact algorithm**, not an approximation. The math is identical to the standard formula:
$$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
The speedup comes from changing **how** that computation runs on the GPU, not **what** it computes. Naive attention writes the full n-by-n matrix `S = QK^T` to HBM, reads it back for softmax, materializes the n-by-n probability matrix, reads it again for the matmul with V. That is roughly 4 round trips through HBM, and the n-by-n matrix is the dominant memory cost at long sequence length.
FlashAttention tiles Q, K, V into SRAM-sized blocks. Each tile pair runs the partial softmax with the online-softmax update rule (track a running max and a running exp-sum so that softmax over blocks fuses into one global softmax). The full n-by-n matrix never exists in HBM at any point. Output is written once.
Result: same numbers, 2-4x faster, and memory drops from O(n^2) to O(n) for the attention buffer. Compute complexity is still O(n^2); only HBM traffic shrinks.
Quick reference
TL;DR
FlashAttention is an exact algorithm. Same output as naive attention up to floating-point reduction order; only the HBM and SRAM traffic pattern changes.
Imagine you have to add up a million numbers and your scratchpad only fits a few hundred at a time. The slow way is to keep running back to a giant whiteboard, copying chunks in and partial sums out, until you finish. The fast way is to load one batch into your scratchpad, finish all the math you can on it, then load the next batch, and combine results in your head as you go. You get the exact same total either way, what changes is how much you walked to the whiteboard. FlashAttention is the fast way for attention: same answer, far less walking.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very short sequences (n < 256) where naive attention already fits in SRAM The tiling overhead and kernel-launch cost can exceed the I/O savings when the full n-by-n matrix is tiny.
Hardware without sufficient SRAM per SM (older or embedded GPUs) FlashAttention's tile sizes assume tens to hundreds of KB of on-chip SRAM; very small caches force tiny tiles and the win shrinks.
Custom attention variants with non-standard masking or score modifications not covered by the kernel FlashAttention kernels target standard scaled dot-product attention with common masks (causal, padding); unusual operations may need to fall back to the naive math path.
Workloads that need direct access to the n-by-n attention matrix for interpretability or analysis FlashAttention never materializes the full attention matrix, so downstream tools that scan attention weights cannot run on top of it; you need the naive path for that.
Quick scan · 2–4 min
Everything important, quickly.
60s: state FlashAttention is exact (not approximate), show the standard softmax(QK^T/sqrt(d_k))V formula, explain tiling + online softmax means the n-by-n matrix never lives in HBM, distinguish from Performer/Linformer/sparse attention, note compute stays O(n^2) but memory drops to O(n).
Real-world examples
Real products, models, and research that use this idea.
1PyTorch torch.nn.functional.scaled_dot_product_attention dispatches to FlashAttention v2 on supported GPUs as of PyTorch 2.x.
2Llama 4 Maverick training and inference pipelines use FlashAttention v3 as the default attention kernel on Hopper hardware.
3Hugging Face transformers auto-enables FlashAttention via the attn_implementation flag for any model that defines compatible attention layers.
4Mistral Large 3 and DeepSeek V4 production serving stacks rely on FlashAttention as the base kernel beneath GQA, MLA, and Flash-Decoding extensions.
5Triton and CUDA reference implementations from the tridao/flash-attention repo are the canonical source used by vLLM and TensorRT-LLM.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf FlashAttention is exact, why do training runs sometimes show small numerical differences between FlashAttention and a naive PyTorch attention?
Sign in or go Pro to see the approach for this follow-up.
QWhy is HBM bandwidth, not FLOPs, the binding constraint that FlashAttention targets?
Sign in or go Pro to see the approach for this follow-up.
QHow does the FlashAttention backward pass stay exact without materializing the full attention matrix?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Calling FlashAttention an approximation. It is not. Performer and Linformer approximate; FlashAttention reorganizes the I/O of the exact same softmax(QK^T/sqrt(d_k))V.