Walk through what FlashAttention does differently from standard attention, and what changes between v1, v2, and v3.

**FlashAttention** computes the same output as the textbook formula: $$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ The textbook version builds an n×n attention matrix as an intermediate, and that matrix lives in HBM, the GPU's main memory. Writing and reading it is slow compared to the tensor core math; most of attention's runtime is spent moving bytes, not computing. FlashAttention fixes this with two ideas. **Tiling**: split Q into row blocks and K, V into column blocks; for each Q-block, loop over K, V blocks and do all the math for that pair in fast on chip SRAM. **Online softmax**: track a running max `m` and running denominator `l` per row as new K-blocks arrive, rescaling the partial output so the final result matches a full row softmax exactly. The n×n matrix never reaches HBM, so the slow round trips disappear. You get 2-4x speedup and memory drops from `O(n²)` to `O(n)`. The math is identical; FlashAttention is NOT an approximation, only FP rounding differs. v2 (2023) reworks parallelism so the algorithm uses SMs better at long context with small batch. v3 (2024) targets H100 with async TMA copies and FP8 tensor cores.

Standard attention writes the n×n score matrix to HBM, reads it back twice, and those round trips dominate runtime because attention is bandwidth bound rather than compute bound. On A100, the arithmetic intensity (~150 FLOPs per HBM byte) is high enough that any kernel touching HBM more than necessary leaves the tensor cores idle. **FlashAttention** tiles Q, K, V into SRAM-sized blocks. An online softmax with a running max and running denominator lets the algorithm compute attention block by block and produce a final result identical to a single pass softmax. The n×n matrix is never materialized in HBM. Math is exact (up to FP reduction order), memory drops from `O(n²)` to `O(n)`, and wall clock drops 2-4x. The backward pass recomputes attention from saved `Q, K, V, m, l` rather than storing scores, paying cheap FLOPs to avoid expensive bytes. v2 (2023) reorders the loops for sequence dim parallelism, which is the regime long context training spends most of its time in, and halves non-matmul work. v3 (2024) targets H100: TMA async copies overlap HBM-SMEM traffic with compute, warpgroup MMA gives async tensor core ops, and FP8 (E4M3, E5M2) delivers another 1.5-2x.