Does FlashAttention change the attention output relative to standard attention?

**FlashAttention** is an exact algorithm. It produces the same output as the textbook attention formula, the difference is purely in how the computation is scheduled on a GPU. The naive implementation builds the full attention matrix `S = QK^T` in HBM (high-bandwidth memory), runs softmax over rows, multiplies by V, and writes the result back. For sequence length n that matrix has `n * n` entries, which is the dominant memory cost and also the main source of HBM bandwidth pressure. FlashAttention reorganizes the work so that the n-by-n matrix never lives in HBM. It tiles Q, K, and V into blocks small enough to fit in the GPU's on-chip SRAM (a few hundred KB per SM), runs attention block by block using an online softmax that updates a running maximum and a running denominator as new blocks arrive, and writes only the final output back to HBM. The key word is **exact**. Not 'good enough', not 'approximately the same', the same answer up to the floating-point order of summation. That is why FlashAttention shipped as a drop-in replacement in PyTorch with no model-quality regression.

**FlashAttention is an exact algorithm**, not an approximation. The math is identical to the standard formula: $$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ The speedup comes from changing **how** that computation runs on the GPU, not **what** it computes. Naive attention writes the full n-by-n matrix `S = QK^T` to HBM, reads it back for softmax, materializes the n-by-n probability matrix, reads it again for the matmul with V. That is roughly 4 round trips through HBM, and the n-by-n matrix is the dominant memory cost at long sequence length. FlashAttention tiles Q, K, V into SRAM-sized blocks. Each tile pair runs the partial softmax with the online-softmax update rule (track a running max and a running exp-sum so that softmax over blocks fuses into one global softmax). The full n-by-n matrix never exists in HBM at any point. Output is written once. Result: same numbers, 2-4x faster, and memory drops from O(n^2) to O(n) for the attention buffer. Compute complexity is still O(n^2); only HBM traffic shrinks.