Why is batching the single biggest throughput lever for LLM decode?

During decode, the model generates one token at a time. To produce that single token, it must read every weight matrix in the network from GPU memory (HBM) and do a small amount of math with each. The reading is slow; the math is fast. So a single request spends most of its time waiting on memory, not computing. This is called being **memory-bandwidth-bound**. The arithmetic intensity, meaning the FLOPs done per byte read, is tiny at batch 1. The expensive HBM trip feeds just one token. Batching fixes the waste. When you process many requests' next tokens in the same step, you read each weight matrix once and reuse it for all of them. The slow memory trip is paid once and shared across the whole batch. Throughput rises almost linearly with batch size, because you generate more tokens for nearly the same memory cost. This keeps going until the GPU's compute units saturate. After that point, adding more requests stops helping throughput and just makes each request wait longer. A CNN classifier is already compute-bound at batch 1, so it has little idle compute to reclaim. That is why batching gives huge wins for language model decode but only modest wins for a vision model.

**Batching** is the biggest decode throughput lever because decode is memory-bandwidth-bound at batch 1. Each step reads the full weight matrix from HBM and does very little math per byte. Arithmetic intensity sits near one. Batching N requests pays that weight read ONCE per step. It amortizes the read across N tokens generated together. Intensity climbs linearly with batch, so the workload slides up the bandwidth slope of the roofline. Throughput rises almost linearly with N. The scaling breaks at the critical batch. Call it B-star. There intensity meets the compute roof. B-star roughly equals peak FLOPs divided by peak bandwidth, often a few hundred to about a thousand tokens. Past B-star the GPU is compute-saturated. Throughput plateaus, and per-token latency grows linearly because requests now queue behind a busy GPU. The contrast with a CNN is the point. A CNN classifier is already compute-bound at batch 1, so it has little idle compute to reclaim. Batching buys only modest gains there. Decode starts deep in the memory-bound regime, so it can see a tenfold throughput jump or more.