Compute the critical batch size where decode crosses from memory bound to compute bound on H100
H100 SXM specifications (FP16, dense, rounded for easy arithmetic): - Peak compute: ~989 TFLOPS (989 × 10^12 FLOP/s) - Peak HBM bandwidth: ~3 TB/s (3 × 10^12 B/s; rated 3.35 TB/s) For LLM decode, arithmetic intensity ≈ batch size B (the weight matrix is read once per step and shared by all B tokens, and each token does ~2N FLOPs of math with it). Compute the critical batch B* where decode crosses from memory bandwidth bound to compute bound on this hardware. Round to the nearest integer. State the regime for B = 100 vs B = 500.
The ridge point is peak FLOPs over peak bandwidth: 989 / 3 ≈ 330. Decode arithmetic intensity tracks batch size, so B* ≈ 330, meaning 100 is memory-bound and 500 is compute-bound.
Picture a kitchen where one chef chops ingredients incredibly fast, but a single narrow doorway delivers groceries from the truck. If only a few dishes are cooking, the chef sits idle waiting for deliveries through that doorway. That is the memory-bound regime: the doorway, not the chef, sets the pace. As you cook more dishes at once from the same delivered groceries, the chef finally has enough to stay busy. Past some number of simultaneous dishes, the chef becomes the limit instead of the doorway. That tipping number is the ridge point. On an H100, with its bandwidth rounded to 3 TB/s, the chef is fast enough that you need roughly 330 dishes batched together before the chef, not the doorway, becomes the bottleneck.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Finding the critical decode batch is a roofline problem. The roofline model says any kernel on a given chip is bounded either by how fast the chip does math or by how fast it moves data, and a single ratio tells you which. The boundary between the two regimes is the ridge point, and on data-center GPUs it sits surprisingly high.
What makes LLM decode special is that the weights are read once from HBM and then reused across every token in the batch. A larger batch does more arithmetic per byte of weights moved, so arithmetic intensity climbs in proportion to batch size. That turns an abstract hardware ratio into a concrete, actionable number: the batch size at which you stop being limited by memory bandwidth and start being limited by raw compute.
On an H100 SXM, with bandwidth rounded to 3 TB/s, the division of 989 by 3 gives about 330. Two corrections matter in production: the KV cache pushes the crossover to a larger batch, and the ratio differs from chip to chip. Each side of the line also calls for a different throughput lever.
The roofline model and the ridge point
The roofline model plots achievable performance against arithmetic intensity, the number of FLOPs a kernel performs per byte it reads from memory. Two ceilings bound it. A sloped ceiling rises with intensity and represents the memory system: at low intensity you are limited by bandwidth. A flat ceiling represents peak compute: at high intensity you are limited by the math units.
The attainable performance is the minimum of those two ceilings. At low intensity the achievable rate is intensity multiplied by bandwidth, a straight line through the origin. As intensity grows the line eventually hits the flat compute ceiling and stops rising. Performance is whichever ceiling is lower at your operating point, which is why the model is so useful: you read off the regime by eye.
The two ceilings meet at the ridge point. Left of it a kernel is memory-bound, because it asks for many bytes relative to the math it does, and the memory system cannot feed the compute units fast enough. Right of it a kernel is compute-bound, because it does plenty of math per byte and the math units saturate first.
The ridge point has a clean closed form. It is peak compute divided by peak bandwidth, expressed in FLOPs per byte. That single ratio characterizes the hardware, and it is the number every kernel's intensity gets compared against. The same lens explains why dense training matmuls run near peak while single-token decode is slow: the matmuls carry high intensity and live on the right of the ridge, while a batch-one decode step is almost pure memory traffic and lives far to the left.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM continuous batching exists to raise the decode batch toward the ridge-point batch and reclaim idle compute during the memory-bound decode phase on H100 and B200.
- NVIDIA's published H100 SXM specs (about 989 dense FP16 TFLOPS, 3.35 TB/s HBM3) put the weight-only ridge point near 295 FLOPs per byte.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the KV cache push the effective crossover above the weight-only B* of 330?
The weight-only ratio assumes the only HBM traffic is the shared weight read. Decode also streams each sequence's KV cache every step, adding bytes that grow with batch while the attention math stays small. Lower intensity at every batch means you need a larger batch to reach the compute roof, and at long context you may never reach it.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating B* as a fixed law of nature. It is a per-GPU hardware ratio, and KV-cache reads push the effective crossover to a larger batch than the weight-only number suggests.
60 second bullets to scan on the way to the call.
The roofline ridge point as peak compute divided by peak bandwidth in FLOPs per byte
Why decode arithmetic intensity scales with batch size B
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.