Walk through the byte accounting that proves a single-batch decode step is bandwidth-bound on H100.

Attempt it

Use H100 numbers (3 TB/s HBM, 989 TFLOPS FP16) to prove that a single-batch Llama-70B decode step is bandwidth-bound rather than compute-bound. Show the byte accounting and the time each phase takes.

Free · 2 AI evals / day

Quick reference

TL;DR

At batch 1 each decode step reads all 140 GB of FP16 weights once for tiny GEMV compute, so latency tracks bytes over bandwidth, roughly 47 ms on H100.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a chef who can chop in a blink but keeps every recipe in a giant cookbook locked in a back room. To cook one dish the chef must walk to the back room, haul out the whole cookbook, read a single line, then cook. The walking and carrying takes minutes; the cooking takes a second. The bottleneck is the trip, not the cooking. A decode step is the same. The GPU must drag every model weight out of memory just to produce one token. The actual arithmetic is trivial, so the time is set entirely by how fast the memory bus moves the weights. Cooking for one customer or twenty takes the same single trip, which is why serving many requests together is so much cheaper per dish.

Key concepts

Memory-bandwidth-bound Arithmetic intensity GEMV decode step Batching

Quick scan · 2–4 min

Everything important, quickly.

Walk the byte accounting, both divisions, the 300 times ratio, and the operational claim that 47 ms is the floor; then list the three levers with concrete numbers and explain why FLOP-cutting tricks do nothing.

Side-by-side

Lever	Effect on bytes	New floor (70B)	Composes with
FP16 baseline	140 GB read	47 ms	baseline
FP8 quant	70 GB read	23 ms	Batching, TP
INT4 quant	35 GB read	11.7 ms	Batching, TP
Batch B=64	Same 140 GB per step, 64 tokens out	47 ms for 64 tokens	Quant, TP
TP-8	140 GB across 8 HBMs	5.8 ms plus allreduce	Quant, Batching

Real-world examples

Real products, models, and research that use this idea.

vLLM batch 1 Llama 3.1 70B at FP16 measures near 21 tokens per second per request, matching the roughly 47 ms per-step floor this math predicts.
Together AI serves Llama 3.1 70B with INT4 weights plus 8-way tensor parallelism to push past 100 tokens per second per request.
NVIDIA TensorRT-LLM benchmarks publish decode floors for 7B, 13B, and 70B models across FP16, FP8, and INT4 weight formats on H100 and B200.
DeepSeek V4 pairs Multi-head Latent Attention with FP8 weights so the per-step byte read stays low enough for cheap long-context decode.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QRecompute the floor for 7B FP16, 70B INT4, and 70B FP16 with 8-way tensor parallelism.

Sign in or go Pro to see the approach for this follow-up.

QAt what context length does the KV cache term stop being negligible?

Sign in or go Pro to see the approach for this follow-up.

QWhere is the batch size at which decode flips from memory bound to compute bound?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.