Predict the arithmetic intensity of decode at varying batch sizes

Attempt it

Single-token decode on a model with N parameters, FP16 weights (2 bytes / weight).

Ignoring KV cache reads (for a short context this is small relative to weight reads), each decoded token requires:
- Reading the entire weight matrix: 2N bytes from HBM
- Performing one matmul forward pass: ~2N FLOPs per token

With batch size B, ONE weight read serves B parallel token streams; FLOPs scale linearly with B.

Compute the arithmetic intensity (FLOPs / bytes read) for B = 1, 8, 64, and 512. Express each as a plain FLOP/byte number.

Quick reference

TL;DR

Decode arithmetic intensity equals the batch size B, because one shared 2N-byte weight read serves 2N*B FLOPs. At small batch this sits far below the hardware ridge point, so decode is memory-bandwidth-bound.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a chef who must walk to a far pantry and haul out every ingredient before cooking a single dish. The walk is slow and fixed; the cooking is quick. If she makes that long trip for just one plate, she spends almost all her time walking, not cooking. So the kitchen mostly sits idle. Now imagine she cooks eight plates from the same single trip to the pantry. The walk costs the same, but the cooking grows eightfold, and she finally spends real time at the stove. The expensive trip is now shared across many plates. The lesson: when fetching the ingredients dominates and the cooking is tiny, the trick is to serve many plates from one trip, so the slow fetch is paid once and the stove stays busy.

Key concepts

Arithmetic intensity Roofline model Batching Memory-bandwidth bound

Quick scan · 2–4 min

Everything important, quickly.

4 min: define arithmetic intensity, derive AI = B for decode, place it against the H100 ridge point, explain the memory-bound regime, then connect batching and the KV cache ceiling.

Real-world examples

Real products, models, and research that use this idea.

vLLM exists primarily to raise decode batch size via paged attention and continuous batching, pushing arithmetic intensity up toward the H100 ridge point.
NVIDIA's H100 datasheet pairs about 990 BF16 teraFLOPs with 3.35 TB/s HBM3, giving a ridge point near 295 to 330 FLOPs per byte that decode must clear.
Serving Llama 4 or DeepSeek V4 at batch one wastes most of the GPU; production stacks pack hundreds of concurrent streams to amortize the weight read.
TensorRT-LLM on H100 and B200 reports throughput scaling near-linearly with batch until the workload crosses into compute-bound, then flattening.
Anthropic and OpenAI inference economics hinge on this: per-token cost falls roughly with batch until the ridge point, which is why concurrency is the main cost lever.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QAt what batch size does decode cross from memory-bound to compute-bound on an H100?

Sign in or go Pro to see the approach for this follow-up.

QHow does including KV cache reads change the arithmetic intensity calculation?

Sign in or go Pro to see the approach for this follow-up.

QIf the KV cache caps batch below the ridge point, what levers recover throughput?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.