Predict the throughput and per-request latency behaviour as batch size grows past saturation

LLM decoding is **memory-bandwidth-bound** at small batch sizes. Each step reads the whole model from HBM to produce one token per request, which wastes bandwidth when only one request is in flight. Adding more requests to the same step reuses those identical weight reads, so throughput climbs almost linearly while per-request latency barely moves. That free-lunch region ends. As batch grows, the cost per step shifts from reading weights to doing matrix multiplies. Eventually the GPU's compute units run flat out. That crossover is the critical batch, here around batch=128. Past the roof, throughput plateaus. The GPU already does all the FLOPs it can, so batch=256 and batch=512 stay near 7000 tok/s instead of doubling. The early numbers tempt you to extend a straight line, but the line bends over. Meanwhile per-request latency degrades. Each step takes a fixed wall-clock time once compute-bound, yet serves more requests. So time per output token roughly doubles to 37 ms at batch=256 and 73 ms at batch=512. This is exactly why serving systems cap batch size. Past saturation you punish every request for almost no extra throughput.

Past the compute roof, throughput plateaus while per-request latency grows linearly with batch. The measured curve climbs from 80 tok/s at batch=1 to 6400 tok/s at batch=128, roughly 80 times, which is clean bandwidth-bound scaling. The elbow sits at batch=128. My prediction for batch=256 is throughput near 7000 tok/s, only a small gain over 6400. TPOT roughly doubles from 20 ms to about 37 ms. For batch=512 throughput stays near 7000 to 7400 tok/s, and TPOT doubles again to about 73 ms. The mechanism is simple once compute saturates. The GPU does peak FLOPs every step regardless of batch, so total tokens per second is pinned. The same step now serves more requests, so each request waits proportionally longer between its own tokens. Throughput measures GPU output per second; TPOT measures how long one request waits. The operational takeaway is to cap batch slightly below the critical batch. Past it you trade latency for almost zero throughput, so production stacks on H100 typically cap around 256 for 7B class models.