Why is standard attention memory bandwidth bound rather than compute bound on modern GPUs?
Standard attention round trips the n by n score matrix through HBM, and HBM is 10-12x slower than on chip SRAM, so I/O dominates runtime even though tensor cores finish the matmul fast.
Picture a chef (the GPU's compute cores) who can chop vegetables in seconds, but ingredients live in a freezer down the hall (HBM), and only a tiny pile fits on the counter (SRAM). The chef finishes chopping faster than the kitchen runner can fetch new ingredients. The meal speed isn't set by the chef; it's set by how often you keep ingredients on the counter instead of running back to the freezer. FlashAttention is the kitchen rule that says: load a small tile of vegetables onto the counter, do all the chopping there, and only put the finished dish back in the freezer.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Walk the GPU memory hierarchy, compute HBM traffic for the n × n matrix at concrete n, contrast with tensor core arithmetic throughput, explain FlashAttention's online softmax + tiling, and note that the speedup grows with n.
| Memory tier | A100 bandwidth | Size | Used for |
|---|---|---|---|
| Registers | ~ private per thread | Small | Per-thread state |
| SRAM (shared mem) | ~19 TB/s | ~40 MB per GPU | Tile local computation |
| L2 cache | ~5 TB/s | ~40 MB | Cross-SM data sharing |
| HBM (global) | ~1.5 TB/s | 40-80 GB | Weights, activations, intermediate tensors |
Real products, models, and research that use this idea.
- A100 specs: HBM2 bandwidth around 1.5 TB/s, on chip SRAM around 19 TB/s, over a 10x ratio.
- H100 has HBM3 at roughly 3 TB/s, SRAM around 33 TB/s, with higher tensor core throughput; B200 widens the gap further with HBM3e.
- FlashAttention 3 ships with Hopper-specific TMA, fp8, and async tensor core paths; FlashAttention 4 work targets Blackwell.
- vLLM and SGLang both apply memory hierarchy thinking to inference KV management via PagedAttention.
- Training Llama 4 Maverick and Mistral Large 3 at long context relies on FlashAttention to keep score matrices off HBM.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is arithmetic intensity, and where's the memory bound/compute bound threshold on A100?
QHow does FlashAttention's online softmax avoid materializing the full attention matrix?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying attention is slow because of 'too many softmaxes' or 'too many matmuls'. The arithmetic finishes fast on tensor cores; it's the HBM I/O for the n × n matrix that's the bottleneck.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.