Why does batching help LLM decode disproportionately more than batching helps CNN inference?
Explain why a typical CNN classifier gets marginal throughput wins from batching while LLM decode sees ~10× or more. Tie the answer to where each workload sits at batch 1.
LLM decode at batch 1 is memory bound, so batching amortizes one weight read across many requests and throughput climbs nearly linearly. CNNs start compute bound, so batching just queues work.
Picture a chef who must walk to a far pantry, carry one heavy sack of flour back, and bake just one loaf with it. The walk dominates the time. If one customer orders a single loaf, the chef wastes the whole trip on that one loaf. But if ten customers each want a loaf, the chef makes one trip and bakes ten loaves from the same sack. The slow walk is paid once and shared across ten orders, so output jumps almost tenfold with barely more time. Now picture a different cook whose counter is already packed and whose hands never stop moving. Giving that cook ten orders does not help, because the hands were already the bottleneck. LLM decode is the first chef. The pantry walk is reading weights from memory.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Batching gives the biggest throughput gain in LLM serving, yet for a CNN classifier it changes little. The same operation, on the same GPU, gives very different returns. The reason is not a vague intuition that big matrices are fast or that GPUs like parallelism. It is a specific binding constraint, and batching either relaxes it or does not.
The explanation comes from the roofline model and the concept of arithmetic intensity. A workload is bound either by how fast the chip can compute or by how fast it can stream data from high-bandwidth memory. Batching changes a workload's position on that roofline, and the gain depends on where the workload started. If you start at the compute ceiling, there is nowhere to climb. If you start far down the bandwidth slope, there is enormous room to climb. The CNN and the decoder sit on opposite ends of that curve.
The same simple arithmetic works for any new workload. Place it on the roofline at batch 1, and you can predict before benchmarking whether batching is worth the engineering.
The roofline model and arithmetic intensity
The roofline model plots achievable throughput against arithmetic intensity, defined as the floating-point operations performed per byte moved from memory. The plot has two regions. On the left is a rising line whose slope is the memory bandwidth. Here you are memory bound, and throughput is capped by how fast bytes arrive. On the right is a flat ceiling set by the peak compute rate. Here you are compute bound, and faster data movement would not help.
The two regions meet at a single intensity, the ridge point. Its value is fixed by the hardware: it is peak compute divided by peak bandwidth. On an H100 that ratio works out to a few hundred flops per byte, which is the threshold any kernel must clear to be compute bound. A workload below that intensity is memory bound and leaves compute units idle while they wait on memory. A workload above it is compute bound and leaves memory bandwidth underused while the math units run at full speed.
The central question for any optimization is which side of the ridge you are on. Batching, quantization, and fusion all work by moving a workload along this curve. Quantization shrinks the bytes moved, which slides a memory-bound kernel rightward. Fusion removes intermediate reads and writes, which does the same. Batching adds parallel work without adding weight bytes, which is the cleanest rightward move of all. If you do not know where you started, you cannot predict whether a given change helps at all. You can then spend weeks tuning a change that the roofline shows cannot help.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM uses continuous batching to push decode batch size toward the critical point, delivering 5 to 10x throughput over naive request-level serving.
- Serving Llama 3.1 70B in bf16 across H100s, decode at batch 1 reads roughly 140 GB of weights per token in total and is HBM bound. Batch 64 amortizes that read across 64 sequences.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat sets the critical batch size B* where decode crosses from memory bound to compute bound?
It is the intensity where the bandwidth slope meets the compute ceiling: roughly peak flops divided by peak HBM bandwidth. Solve for the batch at which bytes moved per step times that ratio equals the flops being done, accounting for weight reuse.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming batching helps every workload equally, or that it helps because the GPU likes bigger matrices. It only helps when arithmetic intensity has headroom before the compute roof.
60 second bullets to scan on the way to the call.
Where CNN inference and LLM decode each sit on the roofline at batch 1
Why convolution has high arithmetic intensity from kernel reuse
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.