During prefill, attention is compute bound, but during decode, the same layer becomes memory bandwidth bound. Explain why generating one token per step changes the regime, and name two optimizations that exploit this insight.
At decode, Q has length 1 per step, so the attention matmul is tiny, but the full KV cache must stream from HBM, making bandwidth (not FLOPs) the bottleneck.
Imagine a chef cooking one omelette per minute (decode) versus a chef cooking 100 omelettes in a batch (prefill). For the single omelette, the time is dominated by walking to the fridge to fetch all the ingredients (streaming the cache from memory), the actual cooking is a few seconds. For the batch, the chef opens the fridge once, takes out everything, and the cooking dominates. The kitchen bottleneck flips from 'how fast can the chef cook' (compute) to 'how fast can you fetch ingredients' (memory bandwidth). Decode-time LLM serving has the same flip every single token step.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The regime flip between prefill and decode is the single most important performance characteristic of modern LLM serving. Understanding it is the prerequisite for understanding why every decode-side optimization (GQA, MQA, MLA, KV quantization, Flash-Decoding, speculative decoding) attacks bandwidth rather than compute, and why prefill-side optimizations (FlashAttention, tensor-core kernels) are largely orthogonal.
This card walks the arithmetic intensity argument, computes concrete cache sizes for production models, surveys the decode-side optimizations that flip the bandwidth bottleneck, and shows how they compose multiplicatively in the 2026 production stack to deliver 20-50x decode latency improvements over textbook MHA-fp16 baselines.
Prefill vs decode: the query shape determines the regime
The single difference between prefill and decode is the query shape. Everything else flows from that.
Prefill
- The model processes the user's full prompt in one forward pass.
- Q has shape
(T, d)where T is the prompt length. - K and V are computed from the same input, also shape
(T, d). - Attention matmul:
(T, T) = (T, d) . (T, d)^T. FLOPs scale asO(T^2 d). - KV is loaded from HBM once and amortized across all T queries.
- Arithmetic intensity scales as
O(T).
For T = 2000 on H100, intensity is well above the 10-ops/byte balance point. The kernel is compute-bound and tensor cores saturate.
Decode
- The model generates one new token per step.
- Q has shape
(1, d). One new query. - K and V have shape
(T, d), the cached history at position T. - Attention matmul:
(1, T) = (1, d) . (T, d)^T. FLOPs scale asO(T d). - KV must be loaded from HBM EVERY decode step. The cache is NOT amortized.
- Arithmetic intensity scales as
O(1), independent of T.
Intensity is roughly 1 op/byte, well below the balance point. Tensor cores idle while HBM streams.
The roofline interpretation
The GPU roofline model says achievable throughput is min(peak_compute, intensity * peak_bandwidth). Above the balance point, performance scales with compute; below it, performance scales with bandwidth.
Prefill sits above the balance point. Decode sits below. The same attention layer is in different regimes depending on which phase it is running.
The regime flip is structural, not algorithmic. You cannot tune a decode kernel out of memory-bound; you can only reduce the bytes it must move.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | Prefill | Decode |
|---|---|---|
| Q shape per layer per head | (T, d) | (1, d) |
| K, V shape | (T, d), built from input | (T, d), cached |
| FLOPs per layer | O(T^2 d), large | O(T d), small |
| Bytes loaded per layer | O(T d), amortized over T queries | O(T d), per single query |
| Arithmetic intensity | O(T), high | O(1), low |
| Regime | Compute-bound (tensor cores) | Memory bandwidth bound (HBM) |
| Bottleneck | FLOPs | Bytes streamed per step |
| Main optimizations | FlashAttention, tensor-core efficient kernels | GQA/MQA/MLA, KV quantization, Flash-Decoding |
Real products, models, and research that use this idea.
- Llama 4 Maverick uses GQA with grouped K and V heads to cut decode-time KV bandwidth, the standard production technique for long-context serving.
- DeepSeek V4's MLA compresses KV into a small latent, achieving roughly 10x cache reduction vs GQA on long contexts.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does FlashAttention not solve the decode bandwidth problem?
FlashAttention reduces HBM round-trips during prefill by tiling Q, K, V into SRAM and computing attention with online softmax. The reduction comes from not materializing the full attention matrix A in HBM, which saves a re-load. At decode, Q has length 1 so there is no A to materialize anyway; the bottleneck is loading K and V themselves, which FlashAttention cannot avoid (you must read every cached entry to compute attention against it). FlashAttention helps prefill; GQA, MLA, and KV quantization help decode.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming decode is slow because of FLOPs. The FLOPs per decode step are tiny; the bottleneck is moving the KV cache from HBM to SRAM, which is why every decode optimization targets bytes per token rather than ops per token.
60 second bullets to scan on the way to the call.
Query shape at decode is (1, d) per layer per head, contrasted with (T, d) at prefill
Arithmetic intensity is O(1) at decode, O(T) at prefill
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.