Why does batching help LLM decode disproportionately more than batching helps CNN inference?

Every GPU operation is limited by one of two things: how fast it can do math (compute), or how fast it can move data from memory (bandwidth). The **roofline model** is the picture that tells you which limit you are hitting for a given workload. A CNN classifying one image is already compute bound. A convolution kernel is reused across every spatial position of the image, so the chip does a lot of math per byte it loads. The math units are already busy. Adding more images to a batch gives the GPU more work but no faster way to do it, so throughput barely improves. LLM decode is the opposite. To generate one token, the model reads its entire weight matrix from memory but does very little math with each weight, because there is only one token's worth of activations. The chip spends almost all its time waiting on memory, not computing. Here is the lever: batching many requests together reuses that single weight read across all of them. You pay the expensive memory read once and compute many tokens from it. Throughput rises almost linearly with batch size until the math units finally fill up.

**The roofline model** explains the gap. Every workload is either compute bound or memory bandwidth bound. Batching only helps the bandwidth-bound case, because only there does extra work fill idle compute units. A CNN at batch 1 is already compute bound. Convolution reuses each kernel weight across many spatial positions, so arithmetic intensity is high. The math units are saturated. Batching more images raises total work but not the rate, since the GPU was already maxed. You just queue more. LLM decode at batch 1 sits on the bandwidth slope, with intensity near one. Generating one token reads every weight from HBM but does one vector-matrix multiply per weight. The chip waits on memory. Batching N requests reads each weight once and reuses it across all N token computations. Arithmetic intensity then climbs roughly linearly with the batch. Throughput climbs with it. This continues until the critical batch size, where compute finally saturates and the workload crosses to compute bound. Past that point extra batch buys latency, not throughput. That headroom is why batching is the single biggest decode throughput lever, and an afterthought for CNNs.