H100 specifications (FP16): - Peak compute: ~989 TFLOPS (989 × 10^12 FLOP/s) - Peak HBM bandwidth: ~3 TB/s (3 × 10^12 B/s) For LLM decode, arithmetic intensity ≈ batch size B (each token reads the shared weight matrix once and does ~2N FLOPs of math). Compute the critical batch B* where decode crosses from memory bandwidth bound to compute bound on this hardware. Round to the nearest integer. State the regime for B = 100 vs B = 500.
The ridge point is peak FLOPs over peak bandwidth: 989 / 3 ≈ 330. Decode arithmetic intensity tracks batch size, so B* ≈ 330, meaning 100 is memory-bound and 500 is compute-bound.
Picture a kitchen where one chef chops ingredients incredibly fast, but a single narrow doorway delivers groceries from the truck. If only a few dishes are cooking, the chef sits idle waiting for deliveries through that doorway. That is the memory-bound regime: the doorway, not the chef, sets the pace. As you cook more dishes at once from the same delivered groceries, the chef finally has enough to stay busy. Past some number of simultaneous dishes, the chef becomes the limit instead of the doorway. That tipping number is the ridge point. On an H100, the chef is fast enough that you need roughly 330 dishes batched together before the chef, not the doorway, becomes the bottleneck.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question is really a roofline problem dressed as a batching question. The roofline model says any kernel on a given chip is bounded either by how fast the chip does math or by how fast it moves data, and a single ratio tells you which. The boundary between the two regimes is the ridge point, and on modern accelerators it sits surprisingly high.
The trick that makes LLM decode special is that the weights are read once from HBM and then reused across every token in the batch. A larger batch does more arithmetic per byte of weights moved, so arithmetic intensity climbs in proportion to batch size. That turns an abstract hardware ratio into a concrete, actionable number: the batch size at which you stop being starved for data and start being limited by raw compute.
The goal of this deep dive is to make the arithmetic second nature, to show why the clean division of 989 over 3 gives about 330, and then to explain the two corrections that separate a textbook answer from a production one: the KV cache pulling the crossover lower, and the ridge point drifting upward every GPU generation. By the end you should be able to size a decode batch on a napkin and explain which throughput lever to reach for on each side of the line.
The roofline model and the ridge point
The roofline model plots achievable performance against arithmetic intensity, the number of FLOPs a kernel performs per byte it reads from memory. Two ceilings bound it. A sloped ceiling rises with intensity and represents the memory system: at low intensity you are limited by bandwidth. A flat ceiling represents peak compute: at high intensity you are limited by the math units.
The attainable performance is the minimum of those two ceilings. At low intensity the achievable rate is intensity multiplied by bandwidth, a straight line through the origin. As intensity grows the line eventually hits the flat compute ceiling and stops rising. Performance is whichever ceiling is lower at your operating point, which is why the model is so useful: you read off the regime by eye.
The two ceilings meet at the ridge point. Left of it a kernel is memory-bound, because it asks for many bytes relative to the math it does, and the memory system cannot feed the compute units fast enough. Right of it a kernel is compute-bound, because it does plenty of math per byte and the math units saturate first.
The ridge point has a clean closed form. It is peak compute divided by peak bandwidth, expressed in FLOPs per byte. That single ratio is the hardware's signature, and it is the number every kernel's intensity gets compared against. The same lens explains why dense training matmuls run near peak while single-token decode crawls: the matmuls carry high intensity and live on the right of the ridge, while a batch-one decode step is almost pure memory traffic and lives far to the left.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM continuous batching exists precisely to climb toward the ridge-point batch and reclaim idle compute during the memory-bound decode phase on H100 and B200.
- NVIDIA's published H100 specs (989 FP16 TFLOPS, ~3.35 TB/s HBM3) make the weight-only ridge point land near 330, the basis of most serving capacity planning in 2026.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the KV cache pull the effective crossover below the weight-only B* of 330?
The weight-only ratio assumes the only HBM traffic is the shared weight read. Decode also streams the per-token KV cache each step, adding bytes that do not scale the math the same way. More bytes per FLOP lowers arithmetic intensity, so you hit the compute roof at a smaller batch.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating B* as a fixed law of nature. It is a hardware ratio that climbs every GPU generation, and KV-cache reads push the effective crossover lower than the weight-only number suggests.
60 second bullets to scan on the way to the call.
The roofline ridge point as peak compute divided by peak bandwidth in FLOPs per byte
Why decode arithmetic intensity scales with batch size B
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.