What is the primary reason batching is so much more effective for LLM decode than for CNN inference?

Every GPU operation is limited by one of two things: how fast it can compute, or how fast it can move data from memory. Which one limits you decides how much batching helps. LLM decode generates one token at a time. At batch size 1, the GPU loads the entire model weights from memory just to produce a single token. The actual math is tiny compared to that memory traffic. So the GPU spends most of its time waiting on memory rather than computing. This is called being **memory-bandwidth-bound**. Batching fixes exactly this waste. If you process sixteen requests together, you load each weight once and reuse it across all sixteen tokens. The expensive memory read is shared, so throughput rises almost linearly with batch size until the compute units finally fill up. A CNN is the opposite. Its convolutions reuse each filter across many positions in an image, so even at batch 1 the GPU is already busy computing. It is compute-bound from the start. Batching a CNN gives only modest gains because there was little idle time to recover. The lesson is that batching pays off in proportion to how memory-starved the workload was to begin with.

The driver is the **bottleneck regime**, not model size. Place each workload on the roofline. LLM decode at batch 1 sits on the bandwidth slope. A CNN sits near the compute ceiling. Decode is memory-bound because it reads the full weight matrix to emit a single token. Arithmetic intensity is roughly one FLOP per byte. The GPU stalls on memory while compute units idle. That idle headroom is exactly what batching reclaims. Batching many requests reads each weight once and reuses it across all their tokens. Intensity rises linearly with batch. Throughput climbs along the bandwidth slope until it hits the compute roof. A CNN starts near that roof. Its convolutions reuse each kernel across many spatial positions, so intensity is already high at batch 1. There is little stall to recover, so batching adds only marginal throughput. The other options miss this point. Parameter count confuses scale with regime. There is no special large-batch tensor core. Continuous batching is real, but it is a scheduling layer on the same physics, not a different mechanism.