Predict the critical batch size where decode crosses from memory-bound to compute-bound on H100

Every GPU has two relevant speed limits: how fast it can do math (peak FLOPs) and how fast it can move data from memory (peak HBM bandwidth). A workload is **memory-bound** when it spends most of its time waiting on data, and compute-bound when it spends most of its time doing arithmetic. The roofline model gives a clean rule. Compute arithmetic intensity, the ratio of FLOPs done per byte read. Compare it to the hardware ratio of peak FLOPs over peak bandwidth. If intensity is below that ratio you are memory-bound; above it you are compute-bound. For LLM decode the weights are read once and shared across the whole batch, so arithmetic intensity rises roughly in proportion to batch size B. That makes the crossover batch a single division. On the H100, peak FLOPs is about 989 TFLOPS and peak bandwidth is about 3 TB/s. Dividing gives about 330. So a decode batch near 330 tokens is the tipping point: below it bandwidth limits you, above it compute limits you. That is why batching is the biggest throughput lever in decode. You are climbing toward 330 to put the idle compute to work.

The crossover is the roofline ridge point. It is peak compute over peak bandwidth: $B^{*} = \frac{\text{peak FLOP/s}}{\text{peak bytes/s}}$ On an H100 that ratio is about 330. Decode reads each weight matrix once and reuses it across all B tokens in the batch. So arithmetic intensity scales roughly with batch size. The ridge point in FLOPs per byte therefore lands at a batch size of about 330 tokens on the H100. The regimes follow directly. At B = 100 you sit below 330, so decode is memory-bandwidth-bound. Throughput rises almost linearly as you add batch, and time per output token stays roughly flat. At B = 500 you sit above 330, so decode is compute-bound. Throughput plateaus, and adding more batch only raises time per output token. Production systems cap batch slightly below B*. The KV cache must also stream from HBM each step, which the weight-only ratio ignores. That extra traffic pulls the effective crossover down, so serving stacks target roughly 256 to 300 to keep latency predictable.