Decode reads the whole model to emit one token: what does that imply for batching?
Explain why the decode phase is memory bandwidth bound rather than compute bound, and what that implies for how you should batch decode work.
Decode reads all the weights from memory to emit one token, so it's bandwidth-limited — batching many sequences reuses that read and turns idle compute into throughput.
Imagine a chef who has to walk to the far pantry and carry back every ingredient just to cook one tiny dish. The walking takes far longer than the cooking, so the kitchen sits mostly idle. The fix isn't a faster stove — it's cooking many dishes from that one pantry trip. The chef already hauled the ingredients, so adding more plates barely costs extra time. That's batching: many requests share the same expensive trip to memory, and the kitchen finally runs busy.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
"Just buy a faster GPU" is the wrong instinct for decode latency, and watching a candidate reach for it tells an interviewer they have not internalized where the bottleneck lives. Decode is slow not because the chip can't compute fast enough, but because it spends most of its time waiting for bytes to arrive from memory.
That single fact reorganizes everything about how you serve a model. It explains why a state of the art accelerator can sit at single digit percent compute utilization while decoding one sequence, why throughput and latency pull in opposite directions, and why the entire modern serving stack is built around keeping a big batch full rather than making any one request fast.
This deep dive starts from arithmetic intensity and the roofline, derives why batching is nearly free on the dominant cost, then surfaces the real constraint — KV-cache memory — and the production machinery that exists to manage it. The goal is to move you from reciting "decode is memory-bound" to reasoning from it.
Arithmetic intensity: bytes moved versus math done
Every operation on a GPU sits somewhere on a spectrum between compute-bound and memory-bound, and where it sits is decided by its arithmetic intensity — the number of FLOPs performed per byte read from memory.
A decode step is a brutal example of the low-intensity end. To produce one token, the model must read every weight in the network out of HBM. For a 70B-parameter model in 16-bit, that is roughly 140 GB of reads. The arithmetic it does with those weights for a single token is a matrix-vector multiply — about 2 FLOPs per parameter. So you move 140 GB and do roughly 140 GFLOPs: a tiny amount of math per byte.
Contrast prefill, which processes many prompt tokens against the same weights at once. That reuses each weight read across all prompt positions, giving high intensity and saturating the compute units. The same hardware, the same weights — but prefill is compute-bound and decode is memory-bound, purely because of how many useful FLOPs ride along with each byte read.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Rough arithmetic-intensity check for a decode step.
# Decode is memory-bound when intensity sits below the GPU ridge point.
weights_bytes = 70e9 * 2 # 70B params, fp16 -> ~140 GB read per step
bw = 3.3e12 # ~3.3 TB/s HBM bandwidth
flops_per_param_per_token = 2 # one matrix-vector multiply
ridge_point = 295 # GPU peak_flops / bw, FLOPs per byte
def decode_step_ms(batch):
# Weight read is shared across the whole batch -> fixed cost.
return weights_bytes / bw * 1e3
def intensity(batch):
return flops_per_param_per_token * batch # FLOPs per byte read
for B in (1, 8, 32, 256):
bound = "memory-bound" if intensity(B) < ridge_point else "compute-bound"
tok_per_s = B / (decode_step_ms(B) / 1e3)
print(f"B={B:4d} intensity={intensity(B):4d} {bound:12s} {tok_per_s:7.0f} tok/s")
# B=1 stays memory-bound: same ~42 ms read yields 1 token; B=32 yields 32.| Lever | Effect on throughput | Effect on single-sequence latency |
|---|---|---|
| Larger decode batch (B↑) | Rises ~linearly until compute or KV limit | Unchanged or slightly worse |
| Faster-compute GPU, same bandwidth | Little change at B=1 (memory-bound) | Little change |
| Higher memory bandwidth | Rises | Improves per-token latency |
Real products, models, and research that use this idea.
- vLLM combines continuous batching with PagedAttention to keep decode batches large without KV fragmentation.
- NVIDIA TensorRT-LLM uses in-flight batching to amortize weight reads across concurrent decode streams.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the roofline model predict the batch size where decode becomes compute-bound?
Compare arithmetic intensity 2·B FLOPs/byte against the GPU ridge point and solve for the crossover batch size.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking a bigger or faster-compute GPU fixes decode latency, when the bottleneck is memory bandwidth, not FLOPs — so the extra compute sits idle.
60 second bullets to scan on the way to the call.
Definition of arithmetic intensity and the roofline ridge point
Why a single decode step reads all weights for one token
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.