Llama-70B inference at batch 1, FP16, on a single H100. H100 numbers (FP16): - HBM bandwidth: 3 TB/s = 3 × 10^12 B/s - Peak compute: 989 TFLOPS = 989 × 10^12 FLOP/s Per single decoded token: - Weight read from HBM: ~140 GB (70B params × 2 bytes/param) - KV cache read: ignore for this exercise (small at short context) - Forward pass FLOPs: ~140 GFLOPs (2 × N FLOPs / token at fp16) Compute: 1. Weight read time (ms) 2. Compute time (ms) 3. Which dominates, and by how many orders of magnitude?
Reading 140 GB of weights at 3 TB/s takes ~47 ms, while 140 GFLOPs at 989 TFLOPS takes ~0.14 ms, so batch-1 decode is bandwidth-bound by ~300x.
Imagine a chef who must read an enormous 140-page recipe book cover to cover before cooking each single dish. Reading the whole book takes about 47 seconds; the actual chopping and stirring takes a fraction of a second. The dish is slow not because cooking is hard, but because fetching the instructions is slow. A GPU decoding one token works the same way. It must stream all 140 GB of model weights out of memory to produce one token, and that streaming dominates. The arithmetic itself is trivially fast. So if you want faster cooking, you do not buy a faster knife. You find a way to read fewer pages, or cook many dishes from one reading of the book.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This problem is the single most important back-of-envelope calculation in LLM serving. It explains, in two divisions, why decode is slow, why every optimization targets memory rather than math, and why a GPU rated at nearly a petaFLOP can feel sluggish generating one token at a time.
The setup is a 70B-parameter model in FP16 on one H100 at batch 1. To emit a single token, the GPU streams all the weights out of HBM, the high-bandwidth memory stacked beside the compute die. That is roughly 140 GB of traffic per token, because 70 billion parameters at 2 bytes each is 140 billion bytes. We then ask a simple question: how long does that read take, and how long does the actual arithmetic take? The gap between those two answers is the whole story of decode optimization.
By the end you should be able to reproduce both numbers without a calculator, state the ratio, place the workload on a roofline plot, and name the four levers that follow directly from the answer. This is the calculation that separates a candidate who has read about inference from one who has actually profiled it. Interviewers at serving-heavy companies use exactly this problem as a filter, because the answer cannot be memorized as a fact, only derived from understanding what the hardware is doing.
The two divisions, with units
Two quantities matter. Time to read the weights, and time to do the math. Each is a single division, and the units make the answer unambiguous.
Weight-read time is bytes divided by bandwidth. Bytes over bytes-per-second leaves seconds:
Compute time is FLOPs divided by peak FLOPS. FLOPs over FLOPs-per-second again leaves seconds:
The single most common error is mixing the units, dividing bytes by FLOPS or FLOPs by bandwidth. The result is a number with no physical meaning. Keep memory traffic with bandwidth, and compute work with compute rate, and the seconds fall out cleanly.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM and SGLang use continuous batching precisely to amortize the single weight read across many concurrent requests, lifting decode throughput 5 to 10x.
- FP8 KV cache and FP8 weights on H100 and B200, shipped in TensorRT-LLM, cut the bytes-per-token term that this calculation shows is the bottleneck.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does batching change this calculation, and why does it raise throughput but not lower per-token latency?
One weight read of 140 GB now serves B tokens. Bandwidth cost per token drops by B, but the single read still takes 47 ms, so latency per token is unchanged while tokens per second scales. Arithmetic intensity climbs toward the roofline ridge.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating decode latency as a compute problem and reaching for faster matmuls. At batch 1 the GPU is starved on memory bandwidth, so compute speed is a rounding error.
60 second bullets to scan on the way to the call.
The two divisions: bytes over bandwidth, and FLOPs over peak FLOPS
Why each result has units of seconds and how to convert to milliseconds
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.