GPUs have a **memory hierarchy** with several tiers at very different speeds. On A100: HBM (the 40-80 GB in `nvidia-smi`) runs ~1.5 TB/s. On-chip SRAM (registers + shared memory per SM) runs ~19 TB/s, about 12x faster than HBM. H100 widens both ends; B200 with HBM3e widens further.
Standard attention does three big tensor ops: `Q @ K^T` produces an `n × n` score matrix, softmax normalizes rows, and the result times V gives output. Each intermediate lives in HBM. The score matrix is written, read back for softmax, written, then read again for the V multiplication.
For `n = 8192` in fp16, the score matrix is ~128 MB per head. Across many heads and 80 layers, a forward pass moves gigabytes through HBM.
Meanwhile the arithmetic finishes fast. A100 tensor cores sustain ~312 TFLOPS. Compute completes long before the I/O does. Runtime is **memory bandwidth bound**.
**FlashAttention** avoids materializing the `n × n` matrix. Tile Q, K, V into SRAM-sized blocks, compute attention per tile, use online softmax to combine partial results without ever seeing the full matrix. Same math, `O(n)` memory, 2-4x wall clock speedup growing with sequence length.
GPU HBM bandwidth is roughly 1.5 TB/s on A100, while on chip SRAM runs around 19 TB/s, a 10-12x gap. H100 widens both ends with HBM3 at ~3 TB/s and SRAM around 33 TB/s; B200 widens it further with HBM3e.
**Standard attention** writes the full `n × n` attention matrix to HBM after `Q @ K^T` and reads it back twice: once for softmax, once for the V multiplication. For `n = 8192` with 32 heads in fp16, that intermediate is ~4 GB per attention layer. Across 32-80 transformer layers, a forward pass moves hundreds of GB through HBM.
The arithmetic finishes fast on tensor cores (~312 TFLOPS fp16 on A100, ~989 on H100); the I/O does not. Attention is therefore **memory bandwidth bound**, not compute bound.
**FlashAttention** exploits this by tiling Q, K, V in SRAM-sized blocks (typically 128 × 64) and using online softmax to combine partial results without ever materializing the full score matrix. Same math, `O(n)` memory instead of `O(n²)`, 2-4x wall clock speedup that grows with sequence length and reaches 10x+ at 128k context.
Quick reference
TL;DR
Standard attention round trips the n by n score matrix through HBM, and HBM is 10-12x slower than on chip SRAM, so I/O dominates runtime even though tensor cores finish the matmul fast.
Picture a chef (the GPU's compute cores) who can chop vegetables in seconds, but ingredients live in a freezer down the hall (HBM), and only a tiny pile fits on the counter (SRAM). The chef finishes chopping faster than the kitchen runner can fetch new ingredients. The meal speed isn't set by the chef; it's set by how often you keep ingredients on the counter instead of running back to the freezer. FlashAttention is the kitchen rule that says: load a small tile of vegetables onto the counter, do all the chopping there, and only put the finished dish back in the freezer.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very short sequences such as n below 512 with small heads The n by n matrix fits comfortably in SRAM-adjacent caches; FlashAttention's tiling overhead can outweigh the I/O savings.
Decode time single token attention with a long KV cache FlashAttention targets prefill style square matmuls; decode is bandwidth bound on KV reads and wants FlashDecoding or PagedAttention instead.
Sliding window or local attention workloads with tiny windows The window already keeps the score matrix small in SRAM, so the IO-aware trick has nothing to recover.
Architectures using non-softmax attention such as linear or kernelized variants Online softmax accumulation does not map cleanly; the canonical FlashAttention algorithm targets exact softmax attention.
Hardware without sufficient SRAM per SM such as older or low end accelerators Tile sizes must shrink, the kernel falls back toward standard attention's I/O pattern, and speedups evaporate.
Quick scan · 2–4 min
Everything important, quickly.
Walk the GPU memory hierarchy, compute HBM traffic for the n × n matrix at concrete n, contrast with tensor core arithmetic throughput, explain FlashAttention's online softmax + tiling, and note that the speedup grows with n.
Side-by-side
Memory tier
A100 bandwidth
Size
Used for
Registers
~ private per thread
Small
Per-thread state
SRAM (shared mem)
~19 TB/s
~40 MB per GPU
Tile local computation
L2 cache
~5 TB/s
~40 MB
Cross-SM data sharing
HBM (global)
~1.5 TB/s
40-80 GB
Weights, activations, intermediate tensors
Real-world examples
Real products, models, and research that use this idea.
1A100 specs: HBM2 bandwidth around 1.5 TB/s, on chip SRAM around 19 TB/s, over a 10x ratio.
2H100 has HBM3 at roughly 3 TB/s, SRAM around 33 TB/s, with higher tensor core throughput; B200 widens the gap further with HBM3e.
3FlashAttention 3 ships with Hopper-specific TMA, fp8, and async tensor core paths; FlashAttention 4 work targets Blackwell.
4vLLM and SGLang both apply memory hierarchy thinking to inference KV management via PagedAttention.
5Training Llama 4 Maverick and Mistral Large 3 at long context relies on FlashAttention to keep score matrices off HBM.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is arithmetic intensity, and where's the memory bound/compute bound threshold on A100?
Sign in or go Pro to see the approach for this follow-up.
QHow does FlashAttention's online softmax avoid materializing the full attention matrix?
Sign in or go Pro to see the approach for this follow-up.
QWhy does FlashAttention's speedup grow with sequence length?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying attention is slow because of 'too many softmaxes' or 'too many matmuls'. The arithmetic finishes fast on tensor cores; it's the HBM I/O for the n × n matrix that's the bottleneck.