Decode-phase attention is memory bandwidth bound. Explain what flips the regime.

Short answer·Medium·4.0 · 0·~3 min·Relevant atNVIDIA

Attempt it

During prefill, attention is compute-bound, but during decode, the same layer becomes memory bandwidth bound. Explain why generating one token per step changes the regime, and name two optimizations that exploit this insight.

Free · 2 AI evals / day

**The regime flip comes from the query shape.** ### Prefill: compute-bound During prefill the model processes the user's full prompt in one forward pass. For a 2000-token prompt: - Q has shape `(2000, d)` per layer per head. - K has shape `(2000, d)`. V has shape `(2000, d)`. - The attention matmul `Q K^T` is `(2000, 2000)`, lots of FLOPs. - The KV is loaded from HBM once and amortized across 2000 queries. Arithmetic intensity (FLOPs per byte loaded) is high. Tensor cores saturate. The bottleneck is compute. ### Decode: memory bandwidth bound During decode the model generates one new token per step. For a request at position seq_len = 2000: - Q has shape `(1, d)` per layer per head. ONE query. - K and V have shape `(2000, d)` per layer per head. The full cached history. - The attention matmul `Q K^T` is `(1, 2000)`, tiny FLOPs. - The KV cache must be loaded from HBM EVERY decode step. For a 13B model with a 4k context that is several gigabytes per request, every single step. Bandwidth between HBM and SRAM, not FLOPs, sets the wall-clock cost. ### Two decode-side optimizations **(1) GQA and MQA** shrink the KV cache by sharing K and V across query-head groups. GQA with 8 groups cuts the cache 4-8x, proportionally cutting the bandwidth bill. Used in Llama 3 and 4, Mistral, Qwen. **(2) KV cache quantization** to INT8 or FP8 halves the bytes you have to move per step. Combines with GQA for a multiplicative win. **Bonus**: Flash-Decoding parallelizes the key axis so multiple SMs share the bandwidth load.

**The regime flips because of the query shape.** At decode there is exactly one new query per layer: Q has shape `(batch, n_heads, 1, head_dim)`. The matrix multiply against the cached keys is tiny by FLOP count, modern GPUs can finish it in microseconds of compute. The catch is the operand. To run that multiply you must **stream every cached key (and later every value) from HBM into the GPU's compute units**, layer by layer, head by head. For a 13B model with a 4k context that is several gigabytes per request, every single step. **Bandwidth, not FLOPs, sets the wall-clock cost.** The arithmetic intensity is roughly: $$\text{intensity} = \frac{\text{FLOPs}}{\text{bytes loaded}} \approx \frac{2 \cdot d}{4 \cdot d \cdot T} = \frac{1}{2T}$$ For any reasonable T (~1000+), this falls well below the H100 balance point (~10 ops/byte), so the kernel is firmly memory-bound. Prefill is the mirror image: many queries amortize the same KV load, so the same matmul becomes compute-heavy and the GPU goes back to being FLOP-bound. **Two decode-side optimizations that exploit this:** 1. **GQA and MQA** shrink the cache by sharing K and V across query-head groups. GQA with 8 KV groups (vs 32 query heads) cuts the cache 4x. MQA shares one K and V across ALL query heads for an 8x cut. Used in Llama 3 and 4, Mistral, Qwen, DeepSeek. 2. **KV quantization** to INT8 or FP8 halves the bytes per value moved per step. Combines multiplicatively with GQA. **Flash-Decoding** adds a third lever: parallelize the key axis across multiple SMs to share the bandwidth load and overcome the single-SM HBM bandwidth ceiling.

Quick reference

TL;DR

At decode, Q has length 1 per step, so the attention matmul is tiny, but the full KV cache must stream from HBM, making bandwidth (not FLOPs) the bottleneck.

Memory aid

Easy to grasp

Imagine a chef cooking one omelette per minute (decode) versus a chef cooking 100 omelettes in a batch (prefill). For the single omelette, the time is dominated by walking to the fridge to fetch all the ingredients (streaming the cache from memory), the actual cooking is a few seconds. For the batch, the chef opens the fridge once, takes out everything, and the cooking dominates. The kitchen bottleneck flips from 'how fast can the chef cook' (compute) to 'how fast can you fetch ingredients' (memory bandwidth). Decode-time LLM serving has the same flip every single token step.

Key concepts

Decode phase KV cache Memory bandwidth Grouped query attention

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

8 min: Q shape difference between prefill and decode, arithmetic intensity calculation, roofline interpretation, concrete cache bytes for production models, GQA and MQA as cache shrinkers, KV quantization, Flash-Decoding parallelization, speculative decoding as a regime-flipper, multiplicative composition in the 2026 stack.

Side-by-side

Property	Prefill	Decode
Q shape per layer per head	(T, d)	(1, d)
K, V shape	(T, d), built from input	(T, d), cached
FLOPs per layer	O(T^2 d), large	O(T d), small
Bytes loaded per layer	O(T d), amortized over T queries	O(T d), per single query
Arithmetic intensity	O(T), high	O(1), low
Regime	Compute-bound (tensor cores)	Memory bandwidth bound (HBM)
Bottleneck	FLOPs	Bytes streamed per step
Main optimizations	FlashAttention, tensor-core efficient kernels	GQA/MQA/MLA, KV quantization, Flash-Decoding

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick uses GQA with grouped K and V heads to cut decode-time KV bandwidth, the standard production technique for long-context serving.
DeepSeek V4's MLA compresses KV into a small latent, achieving roughly 10x cache reduction vs GQA on long contexts.
vLLM and TensorRT-LLM ship INT8 and FP8 KV cache quantization options that combine multiplicatively with GQA for further bandwidth savings.
Flash-Decoding (Dao et al. 2023) parallelizes the key axis across multiple SMs to break past single-SM HBM bandwidth limits on long-context decode.
Speculative decoding with draft models like Llama-3-8B verifying for Llama-3-70B raises decode arithmetic intensity and is now standard in 2026 production serving for GPT-5.5, Claude Opus 4.7, and large open models.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does FlashAttention not solve the decode bandwidth problem?

QHow does speculative decoding raise arithmetic intensity?

QHow do GQA and KV quantization compose multiplicatively for decode-time bandwidth savings?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming decode is slow because of FLOPs. The FLOPs per decode step are tiny; the bottleneck is moving the KV cache from HBM to SRAM, which is why every decode optimization targets bytes per token rather than ops per token.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.