Click any words you think contain an error. Click again to unmark.
Batching helps because decode is memory-bound: at batch 1 the GPU streams weights from HBM and barely computes, so adding requests amortizes one weight load over many tokens.
Picture a delivery truck driving across town to hand over one envelope, then driving back empty. The drive is the expensive part; the envelope weighs nothing. That round trip is loading the model weights from memory. Doing it for a single token wastes the whole journey. Batching is loading the truck with envelopes for many addresses on one trip. The drive costs the same whether you carry one envelope or fifty, so the cost per delivery plummets. LLM decode is exactly this: reading the weights dominates, the math per token is tiny. Pack more requests into one weight read and throughput soars. But a workload that already fills the truck every trip gains nothing from a bigger truck.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This passage is seductive because every sentence is built from true facts arranged into a false conclusion. Batching is indeed powerful. Decode, prefill, and CNN inference are indeed matrix multiplications. The error is the leap from "all matmuls" to "all benefit equally," which ignores the single most important question in inference performance: is this kernel limited by compute or by memory bandwidth?
The whole answer rests on one idea. Batching is not a way to add compute. It is a way to stop wasting memory bandwidth. LLM decode at batch 1 reads the entire model from HBM to produce a single token, then throws nearly all of that bandwidth away because it does so little math per byte read. Batching reclaims that waste by serving many tokens from one weight read.
This deep dive builds the roofline mental model, shows precisely why decode is memory-bound while prefill and CNN inference are not, derives how batching shifts a workload along the roofline, and explains why the "just double the batch" rule is correct for decode and wrong for a ResNet. By the end you should be able to predict, before running a single benchmark, whether batching will help a given workload.
The roofline model: compute roof versus bandwidth slope
The roofline model plots achievable throughput against arithmetic intensity, the ratio of useful FLOPs to bytes moved from memory. It has two regions. A diagonal bandwidth slope where performance is limited by how fast memory can feed the cores, and a flat compute roof where performance is limited by peak FLOPs.
The two meet at the ridge point, whose intensity equals peak compute divided by peak bandwidth. On an H100 with roughly 1000 bf16 TFLOPs and roughly 3.3 TB/s of HBM bandwidth, the ridge sits near 300 FLOPs per byte. Any kernel below that intensity is bandwidth-bound and cannot reach peak compute no matter how fast the cores are.
The practical takeaway: to know whether an optimization will help, first locate the workload on this chart. A bandwidth-bound kernel is starved for data and responds to anything that raises intensity. A compute-bound kernel is already saturated and responds only to more FLOPs or a smaller problem.
The passage's whole argument collapses here. It treats "matrix multiplication" as a single performance class, but matmul shape determines intensity, and intensity determines the regime. A square GEMM and a tall-skinny GEMV are both matmuls, yet they sit on opposite sides of the ridge. The unit of analysis is never the operation name; it is FLOPs per byte. Roofline turns a vague intuition about batching into a number you can compute and act on.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM continuous batching is the headline decode throughput lever; it amortizes weight reads across requests and drives utilization toward the compute roof.
- SGLang batches Llama 4 and Qwen 3 decode steps so a single HBM weight read serves dozens of concurrent sequences.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does batching turn a memory-bound decode step into a compute-bound one?
Track arithmetic intensity. A batch-1 GEMV reads each weight once for one multiply-add. Batching to size b reuses each loaded weight b times, so FLOPs per byte scale with b. Intensity climbs until it crosses the roofline ridge and the kernel becomes compute-bound.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming batching helps by adding more compute or because the workload is compute-bound. Decode is memory-bound at batch 1; batching wins by amortizing one weight load over many tokens, not by adding FLOPs.
60 second bullets to scan on the way to the call.
Why LLM decode is memory bandwidth bound at batch 1
How batching amortizes a single weight load across requests
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.