Why does batching N concurrent requests during decode raise GPU throughput far more than batching does for, say, a CNN classifier? Where does the linear scaling start to break down?
Decode is memory-bound, so one weight read per step is wasted on a single token. Batching amortizes that read across many tokens, raising throughput near-linearly until the compute roof.
Imagine a chef who must walk to a huge pantry and haul out every ingredient just to cook one tiny dish. The walk is the slow part, not the cooking. If only one order is on the ticket, that long walk feeds a single plate. But if twenty orders are waiting, the chef hauls the same ingredients once and plates all twenty in roughly the same trip. The walk to the pantry is reading the model weights from memory. The cooking is the actual math. Because the walk dominates, serving twenty requests together is almost as cheap as serving one. That is why batching is the biggest single throughput win for language model serving. It pays the expensive memory trip once and spreads it across many answers.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Batching is the single biggest throughput lever in LLM serving, and the reason is memory bandwidth rather than raw compute. Autoregressive decode starts deep in the memory-bound corner of the roofline. Batching raises its arithmetic intensity until the near-linear scaling runs out at a clear point. That is why batching helps a language model so much more than a vision model.
A decode step reads the full set of model weights from HBM to generate one token, and does almost no arithmetic per byte read. The GPU's compute units sit idle while the memory bus is saturated. The peak FLOPs number does not matter in this regime, because the bottleneck is how fast bytes arrive, not how fast they can be multiplied. Batching reuses each weight read across many requests and turns that idle compute into real throughput, until it hits a hard compute ceiling. The throughput versus latency tension then decides how large a real deployment lets the batch grow.
Why decode is memory-bandwidth-bound at batch 1
Autoregressive decode generates one token per forward pass. Each pass multiplies a single activation vector against every weight matrix in the network. That is a matrix-vector product, and matrix-vector products have very low arithmetic intensity.
Arithmetic intensity is the number of FLOPs performed per byte read from memory. For a matrix-vector product the matrix is read once and each element does a single multiply-add, so intensity is close to one FLOP per byte. A data-center GPU can do hundreds of FLOPs in the time it takes to read one byte from HBM. So the compute units stall waiting on memory.
A 70B model in 8-bit or 16-bit precision is roughly 70 to 140 gigabytes of weights. At a few terabytes per second of bandwidth, streaming those weights once takes tens of milliseconds. That read happens for every token you emit. The matmul against one activation vector finishes long before the next chunk of weights has arrived. So the tensor cores spend most of their cycles stalled.
At batch 1, decode time per step is set almost entirely by how fast you can stream the weights out of HBM. The arithmetic is nearly free. This is the defining fact of LLM serving. Every major optimization, from quantization to speculative decoding to batching itself, is in some sense a response to it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM continuous batching keeps the decode batch full by scheduling at the token level, delivering several times the throughput of static batching on Llama 4 serving.
- NVIDIA TensorRT-LLM exposes in-flight batching plus a tunable max batch size so operators can sit near the critical batch on H200 and B200 hardware.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you estimate the critical batch B-star for a given model and GPU?
Take the ratio of peak compute throughput to peak memory bandwidth to get the ridge-point intensity. Then translate that intensity into a token count using the model's bytes read per token. The weight matrices read each decode step dominate that count.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming batching helps because the GPU has spare compute. The real reason is amortizing the weight read in a memory-bound regime. Spare compute is the symptom, not the cause.
60 second bullets to scan on the way to the call.
Why decode at batch one is memory bandwidth bound rather than compute bound
How a weight read is amortized across batched tokens in a single step
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.