Explain mechanically why batching N concurrent requests during decode raises GPU throughput far more than batching does for, say, a CNN classifier. Where does the linear scaling start to break down?
Decode is memory-bound, so one weight read per step is wasted on a single token. Batching amortizes that read across many tokens, raising throughput near-linearly until the compute roof.
Imagine a chef who must walk to a huge pantry and haul out every ingredient just to cook one tiny dish. The walk is the slow part, not the cooking. If only one order is on the ticket, that long walk feeds a single plate. But if twenty orders are waiting, the chef hauls the same ingredients once and plates all twenty in roughly the same trip. The walk to the pantry is reading the model weights from memory. The cooking is the actual math. Because the walk dominates, serving twenty requests together is almost as cheap as serving one. That is why batching is the biggest single throughput win for language model serving. It pays the expensive memory trip once and spreads it across many answers.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Batching is the single biggest throughput lever in LLM serving, and the reason is entirely about memory bandwidth rather than raw compute. To answer this question well you have to explain why autoregressive decode starts life deep in the memory-bound corner of the roofline, what batching does to arithmetic intensity, and exactly where the near-linear scaling runs out. Most candidates can recite that batching helps; far fewer can say why it helps so much more for a language model than for a vision model, and that gap is exactly what the interviewer is probing.
The short version: a decode step reads the full set of model weights from HBM to generate one token, and does almost no arithmetic per byte read. That makes the GPU's compute units sit idle while the memory bus is saturated. The accelerator's headline FLOPs number is irrelevant in this regime because the bottleneck is how fast bytes arrive, not how fast they can be multiplied. Batching reuses each weight read across many requests, converting that idle compute into real throughput, until you hit a hard compute ceiling.
This deep dive builds the picture from the bottom up: the matrix-vector versus matrix-matrix distinction, the roofline and arithmetic intensity, the critical batch where scaling breaks, why a CNN behaves completely differently, and how continuous batching turns the theory into production throughput. It also surfaces the throughput versus latency tension that decides how large you actually let the batch grow in a real deployment. By the end you should be able to sketch the roofline on a whiteboard, place a model's critical batch on it, and reason about where a serving team would choose to operate.
Why decode is memory-bandwidth-bound at batch 1
Autoregressive decode generates one token per forward pass. Each pass multiplies a single activation vector against every weight matrix in the network. That is a matrix-vector product, and matrix-vector products have terrible arithmetic intensity.
Arithmetic intensity is the number of FLOPs performed per byte read from memory. For a matrix-vector product the matrix is read once and each element does a single multiply-add, so intensity is close to one FLOP per byte. A modern accelerator can do hundreds of FLOPs in the time it takes to read one byte from HBM, so the compute units stall waiting on memory.
Concretely, a 70B model in low precision is roughly 70 to 140 gigabytes of weights. At a few terabytes per second of bandwidth, just streaming those weights once takes tens of milliseconds, and that read happens for every single token you emit. The matmul against one activation vector finishes long before the next chunk of weights has even arrived, so the tensor cores spend most of their cycles stalled.
The upshot: at batch 1, decode time per step is set almost entirely by how fast you can stream the weights out of HBM. The arithmetic is essentially free. This is the defining fact of LLM serving, and every major optimization, from quantization to speculative decoding to batching itself, is in some sense a response to it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM continuous batching keeps the decode batch full by scheduling at the token level, delivering several times the throughput of static batching on Llama 4 serving.
- NVIDIA TensorRT-LLM exposes in-flight batching plus a tunable max batch size so operators can sit near the critical batch on H200 and B200 hardware.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you estimate the critical batch B-star for a given model and GPU?
Take the ratio of peak compute throughput to peak memory bandwidth to get the ridge-point intensity. Then translate that intensity into a token count using the model's bytes read per token, which is dominated by the weight matrices read each decode step.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming batching helps because the GPU has spare compute. The real reason is amortizing the weight read in a memory-bound regime; spare compute is the symptom, not the cause.
60 second bullets to scan on the way to the call.
Why decode at batch one is memory bandwidth bound rather than compute bound
How a weight read is amortized across batched tokens in a single step
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.