Predict the critical batch size where decode crosses from memory-bound to compute-bound on H100

Attempt it

H100 specifications (FP16):
- Peak compute: ~989 TFLOPS (989 × 10^12 FLOP/s)
- Peak HBM bandwidth: ~3 TB/s (3 × 10^12 B/s)

For LLM decode, arithmetic intensity ≈ batch size B (each token reads the shared weight matrix once and does ~2N FLOPs of math).

Compute the critical batch B* where decode crosses from memory-bandwidth-bound to compute-bound on this hardware. Round to the nearest integer. State the regime for B = 100 vs B = 500.

Quick reference

TL;DR

The ridge point is peak FLOPs over peak bandwidth: 989 / 3 ≈ 330. Decode arithmetic intensity tracks batch size, so B* ≈ 330, meaning 100 is memory-bound and 500 is compute-bound.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a kitchen where one chef chops ingredients incredibly fast, but a single narrow doorway delivers groceries from the truck. If only a few dishes are cooking, the chef sits idle waiting for deliveries through that doorway. That is the memory-bound regime: the doorway, not the chef, sets the pace. As you cook more dishes at once from the same delivered groceries, the chef finally has enough to stay busy. Past some number of simultaneous dishes, the chef becomes the limit instead of the doorway. That tipping number is the ridge point. On an H100, the chef is fast enough that you need roughly 330 dishes batched together before the chef, not the doorway, becomes the bottleneck.

Key concepts

Roofline model Arithmetic intensity Batching Memory-bandwidth bound

Quick scan · 2–4 min

Everything important, quickly.

4 min: roofline ridge point = peak FLOPs / bandwidth + why decode AI scales with batch + the 989/3 ≈ 330 arithmetic + regime split at B=100 vs 500 + KV-cache correction + generational drift A100 to B200.

Real-world examples

Real products, models, and research that use this idea.

vLLM continuous batching exists precisely to climb toward the ridge-point batch and reclaim idle compute during the memory-bound decode phase on H100 and B200.
NVIDIA's published H100 specs (989 FP16 TFLOPS, ~3.35 TB/s HBM3) make the weight-only ridge point land near 330, the basis of most serving capacity planning in 2026.
Serving Llama 3.1 70B on H100s, teams cap decode batch near 256 to 300 because KV-cache reads at long context pull the effective crossover below the theoretical 330.
Speculative decoding in TensorRT-LLM targets the above-B* compute-bound regime where extra batch no longer helps and only cutting work per token improves throughput.
DeepSeek V4's Multi-head Latent Attention shrinks KV traffic, which raises the effective ridge point back toward the weight-only ceiling on bandwidth-limited hardware.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the KV cache pull the effective crossover below the weight-only B* of 330?

Sign in or go Pro to see the approach for this follow-up.

QHow does the ridge-point batch differ between an A100, an H100, and a B200?

Sign in or go Pro to see the approach for this follow-up.

QAbove B*, what levers remain to raise decode throughput?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.