Spot the errors in this 'batching is universal' claim

The passage assumes batching is a universal win because every workload is "just matmuls." That hides the real question: where does each workload sit on the **roofline**? The roofline says a kernel is limited either by how fast the GPU computes or by how fast it reads data from memory. LLM decode generates one token at a time. To produce that token, the GPU reads the entire set of model weights from high-bandwidth memory, then does a tiny amount of math. The math finishes almost instantly; the GPU spends most of its time waiting on the memory read. Decode is memory-bound. This is exactly why batching helps decode so much. The weights only have to be read once per step. If you process 32 requests together, that single weight read serves 32 tokens instead of one. The expensive part is shared, so throughput rises almost linearly until the GPU finally hits its compute limit. A ResNet image classifier is different. At batch 1 it is already compute-bound, with the GPU busy doing real math rather than waiting on memory. Doubling its batch just queues more work behind an already-saturated GPU. The passage's claim that the math "works out the same way" is wrong: the two workloads start in opposite regimes.

**Batching helps because LLM decode is memory-bandwidth-bound, not compute-bound.** At batch 1, each decode step reads the full weight matrix from HBM and does little math. The GPU sits idle waiting on memory. The passage's error is treating this as added compute. Batching does not work by giving the GPU more FLOPs to chew on. It works by amortizing one weight load across many requests. Read the weights once, reuse them for every sequence in the batch. Cost per token falls because the dominant cost, the memory read, is now shared. The right framing is the roofline. A workload below the ridge point has low arithmetic intensity and is bandwidth-bound. Batching raises intensity by reusing loaded weights, sliding the workload up the bandwidth slope toward the compute roof. Once it reaches the roof, further batching stops helping. That is why the three named workloads differ. CNN inference and LLM prefill already have high intensity and sit near the roof at modest batch. They gain little. LLM decode starts far below the roof and gains the most. The "double the batch, double the throughput" rule holds for decode and fails for a ResNet that is already compute-saturated.