Why does batching help LLM decode disproportionately more than batching helps CNN inference?
Explain why a typical CNN classifier gets marginal throughput wins from batching while LLM decode sees ~10× or more. Tie the answer to where each workload sits at batch 1.
LLM decode at batch 1 is memory bound, so batching amortizes one weight read across many requests and throughput climbs nearly linearly. CNNs start compute bound, so batching just queues work.
Picture a chef who must walk to a far pantry, carry one heavy sack of flour back, and bake just one loaf with it. The walk dominates the time. If one customer orders a single loaf, the chef wastes the whole trip on that one loaf. But if ten customers each want a loaf, the chef makes one trip and bakes ten loaves from the same sack. The slow walk is paid once and shared across ten orders, so output jumps almost tenfold with barely more time. Now picture a different cook whose counter is already packed and whose hands never stop moving. Giving that cook ten orders does not help, because the hands were already the bottleneck. LLM decode is the first chef; the pantry walk is reading weights from memory.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Batching is the single biggest throughput lever in LLM serving, yet for a CNN classifier it is almost a non-event. The same operation, on the same GPU, produces wildly different returns. Understanding why is one of the cleanest tests of whether a candidate actually understands hardware bottlenecks rather than reciting that GPUs like parallelism. A strong answer never appeals to vague intuition about big matrices being fast; it points at a specific binding constraint and shows how batching does or does not relax it.
The answer lives entirely in the roofline model and the concept of arithmetic intensity. A workload is bound either by how fast the chip can compute or by how fast it can stream data from high-bandwidth memory. Batching changes a workload's position on that roofline, and the gain you get depends entirely on where the workload started. If you start at the compute ceiling, there is nowhere to climb; if you start far down the bandwidth slope, there is enormous room to climb. The CNN and the decoder sit on opposite ends of that curve, which is the whole story in one sentence.
This deep dive builds the roofline picture, places CNN inference and LLM decode on it, shows exactly how batching moves each one, derives the linear throughput scaling and the critical batch where it stops, and connects all of it to the serving techniques that exist to exploit this effect in production. By the end you should be able to do the napkin math for any new workload and predict, before benchmarking, whether batching is worth the engineering.
The roofline model and arithmetic intensity
The roofline model plots achievable throughput against arithmetic intensity, defined as the floating-point operations performed per byte moved from memory. The plot has two regions. On the left, a rising line whose slope is the memory bandwidth: here you are memory bound, and throughput is capped by how fast bytes arrive. On the right, a flat ceiling set by the peak compute rate: here you are compute bound, and faster data movement would not help.
The two regions meet at a single intensity, the ridge point. Its value is fixed by the hardware: it is peak compute divided by peak bandwidth. On an H100 that ratio works out to a few hundred flops per byte, which is the threshold any kernel must clear to be compute bound. A workload below that intensity is memory bound and leaves compute units idle while they wait on memory. A workload above it is compute bound and leaves memory bandwidth underused while the math units run flat out.
The central question for any optimization is which side of the ridge you are on. Batching, quantization, and fusion all work by moving a workload along this curve. Quantization shrinks the bytes moved, which slides a memory-bound kernel rightward. Fusion removes intermediate reads and writes, which does the same. Batching adds parallel work without adding weight bytes, which is the cleanest rightward move of all. If you do not know where you started, you cannot predict whether a given change helps at all, and you can easily spend weeks tuning a lever that the roofline guarantees is flat.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM uses continuous batching to push decode batch size toward the critical point, delivering 5 to 10x throughput over naive request-level serving in 2026.
- Serving Llama 3.1 70B on an H100, decode at batch 1 reads roughly 140 GB of bf16 weights per token and is HBM bound, while batch 64 amortizes that read across 64 sequences.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat sets the critical batch size B* where decode crosses from memory bound to compute bound?
It is the intensity where the bandwidth slope meets the compute ceiling: roughly peak flops divided by peak HBM bandwidth. Solve for the batch at which bytes moved per step times that ratio equals the flops being done, accounting for weight reuse.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming batching helps every workload equally, or that it helps because the GPU likes bigger matrices. It only helps when arithmetic intensity has headroom before the compute roof.
60 second bullets to scan on the way to the call.
Where CNN inference and LLM decode each sit on the roofline at batch 1
Why convolution has high arithmetic intensity from kernel reuse
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.