During prefill, attention is compute-bound, but during decode, the same layer becomes memory bandwidth bound. Explain why generating one token per step changes the regime, and name two optimizations that exploit this insight.
Free · 2 AI evals / day
**The regime flip comes from the query shape.**
### Prefill: compute-bound
During prefill the model processes the user's full prompt in one forward pass. For a 2000-token prompt:
- Q has shape `(2000, d)` per layer per head.
- K has shape `(2000, d)`. V has shape `(2000, d)`.
- The attention matmul `Q K^T` is `(2000, 2000)`, lots of FLOPs.
- The KV is loaded from HBM once and amortized across 2000 queries.
Arithmetic intensity (FLOPs per byte loaded) is high. Tensor cores saturate. The bottleneck is compute.
### Decode: memory bandwidth bound
During decode the model generates one new token per step. For a request at position seq_len = 2000:
- Q has shape `(1, d)` per layer per head. ONE query.
- K and V have shape `(2000, d)` per layer per head. The full cached history.
- The attention matmul `Q K^T` is `(1, 2000)`, tiny FLOPs.
- The KV cache must be loaded from HBM EVERY decode step.
For a 13B model with a 4k context that is several gigabytes per request, every single step. Bandwidth between HBM and SRAM, not FLOPs, sets the wall-clock cost.
### Two decode-side optimizations
**(1) GQA and MQA** shrink the KV cache by sharing K and V across query-head groups. GQA with 8 groups cuts the cache 4-8x, proportionally cutting the bandwidth bill. Used in Llama 3 and 4, Mistral, Qwen.
**(2) KV cache quantization** to INT8 or FP8 halves the bytes you have to move per step. Combines with GQA for a multiplicative win.
**Bonus**: Flash-Decoding parallelizes the key axis so multiple SMs share the bandwidth load.
**The regime flips because of the query shape.**
At decode there is exactly one new query per layer: Q has shape `(batch, n_heads, 1, head_dim)`. The matrix multiply against the cached keys is tiny by FLOP count, modern GPUs can finish it in microseconds of compute. The catch is the operand. To run that multiply you must **stream every cached key (and later every value) from HBM into the GPU's compute units**, layer by layer, head by head. For a 13B model with a 4k context that is several gigabytes per request, every single step.
**Bandwidth, not FLOPs, sets the wall-clock cost.** The arithmetic intensity is roughly:
$$\text{intensity} = \frac{\text{FLOPs}}{\text{bytes loaded}} \approx \frac{2 \cdot d}{4 \cdot d \cdot T} = \frac{1}{2T}$$
For any reasonable T (~1000+), this falls well below the H100 balance point (~10 ops/byte), so the kernel is firmly memory-bound.
Prefill is the mirror image: many queries amortize the same KV load, so the same matmul becomes compute-heavy and the GPU goes back to being FLOP-bound.
**Two decode-side optimizations that exploit this:**
1. **GQA and MQA** shrink the cache by sharing K and V across query-head groups. GQA with 8 KV groups (vs 32 query heads) cuts the cache 4x. MQA shares one K and V across ALL query heads for an 8x cut. Used in Llama 3 and 4, Mistral, Qwen, DeepSeek.
2. **KV quantization** to INT8 or FP8 halves the bytes per value moved per step. Combines multiplicatively with GQA.
**Flash-Decoding** adds a third lever: parallelize the key axis across multiple SMs to share the bandwidth load and overcome the single-SM HBM bandwidth ceiling.
Quick reference
TL;DR
At decode, Q has length 1 per step, so the attention matmul is tiny, but the full KV cache must stream from HBM, making bandwidth (not FLOPs) the bottleneck.
Imagine a chef cooking one omelette per minute (decode) versus a chef cooking 100 omelettes in a batch (prefill). For the single omelette, the time is dominated by walking to the fridge to fetch all the ingredients (streaming the cache from memory), the actual cooking is a few seconds. For the batch, the chef opens the fridge once, takes out everything, and the cooking dominates. The kitchen bottleneck flips from 'how fast can the chef cook' (compute) to 'how fast can you fetch ingredients' (memory bandwidth). Decode-time LLM serving has the same flip every single token step.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Reasoning about prefill latency Prefill is compute-bound, so cache-bandwidth optimizations like GQA and KV quantization have minor impact on prefill latency. They mostly help decode.
Embedding models that produce one pooled vector per input Embedding models do not decode; they just run prefill and emit a pooled embedding. The bandwidth-bound decode regime does not apply.
Workloads where prefill dominates total latency (very short generations) If the request generates only 5-10 tokens, prefill cost dominates and decode optimizations contribute little. Focus on prefill-side wins instead.
Small models on high-bandwidth accelerators where the KV cache fits in SRAM If the entire cache fits in on-chip SRAM (true only for very small models or very short contexts), there is no HBM streaming and the regime is no longer bandwidth-bound.
Quick scan · 2–4 min
Everything important, quickly.
8 min: Q shape difference between prefill and decode, arithmetic intensity calculation, roofline interpretation, concrete cache bytes for production models, GQA and MQA as cache shrinkers, KV quantization, Flash-Decoding parallelization, speculative decoding as a regime-flipper, multiplicative composition in the 2026 stack.
Side-by-side
Property
Prefill
Decode
Q shape per layer per head
(T, d)
(1, d)
K, V shape
(T, d), built from input
(T, d), cached
FLOPs per layer
O(T^2 d), large
O(T d), small
Bytes loaded per layer
O(T d), amortized over T queries
O(T d), per single query
Arithmetic intensity
O(T), high
O(1), low
Regime
Compute-bound (tensor cores)
Memory bandwidth bound (HBM)
Bottleneck
FLOPs
Bytes streamed per step
Main optimizations
FlashAttention, tensor-core efficient kernels
GQA/MQA/MLA, KV quantization, Flash-Decoding
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick uses GQA with grouped K and V heads to cut decode-time KV bandwidth, the standard production technique for long-context serving.
2DeepSeek V4's MLA compresses KV into a small latent, achieving roughly 10x cache reduction vs GQA on long contexts.
3vLLM and TensorRT-LLM ship INT8 and FP8 KV cache quantization options that combine multiplicatively with GQA for further bandwidth savings.
4Flash-Decoding (Dao et al. 2023) parallelizes the key axis across multiple SMs to break past single-SM HBM bandwidth limits on long-context decode.
5Speculative decoding with draft models like Llama-3-8B verifying for Llama-3-70B raises decode arithmetic intensity and is now standard in 2026 production serving for GPT-5.5, Claude Opus 4.7, and large open models.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does FlashAttention not solve the decode bandwidth problem?
Sign in or go Pro to see the approach for this follow-up.
QHow does speculative decoding raise arithmetic intensity?
Sign in or go Pro to see the approach for this follow-up.
QHow do GQA and KV quantization compose multiplicatively for decode-time bandwidth savings?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming decode is slow because of FLOPs. The FLOPs per decode step are tiny; the bottleneck is moving the KV cache from HBM to SRAM, which is why every decode optimization targets bytes per token rather than ops per token.