Predict the bandwidth-vs-compute latency of a single Llama-70B decode step on H100

Attempt it

Llama-70B inference at batch 1, FP16, on a single H100.

H100 numbers (FP16):
- HBM bandwidth: 3 TB/s = 3 × 10^12 B/s
- Peak compute: 989 TFLOPS = 989 × 10^12 FLOP/s

Per single decoded token:
- Weight read from HBM: ~140 GB (70B params × 2 bytes/param)
- KV cache read: ignore for this exercise (small at short context)
- Forward-pass FLOPs: ~140 GFLOPs (2 × N FLOPs / token at fp16)

Compute:
1. Weight-read time (ms)
2. Compute time (ms)
3. Which dominates, and by how many orders of magnitude?

Quick reference

TL;DR

Reading 140 GB of weights at 3 TB/s takes ~47 ms, while 140 GFLOPs at 989 TFLOPS takes ~0.14 ms, so batch-1 decode is bandwidth-bound by ~300x.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a chef who must read an enormous 140-page recipe book cover to cover before cooking each single dish. Reading the whole book takes about 47 seconds; the actual chopping and stirring takes a fraction of a second. The dish is slow not because cooking is hard, but because fetching the instructions is slow. A GPU decoding one token works the same way. It must stream all 140 GB of model weights out of memory to produce one token, and that streaming dominates. The arithmetic itself is trivially fast. So if you want faster cooking, you do not buy a faster knife. You find a way to read fewer pages, or cook many dishes from one reading of the book.

Key concepts

Memory-bandwidth-bound Arithmetic intensity Roofline model Decode phase

Quick scan · 2–4 min

Everything important, quickly.

4 min: two divisions with units + the roughly 300x ratio + roofline and arithmetic intensity framing + the levers (quantization, batching, speculative decoding) + when the KV cache term joins in.

Real-world examples

Real products, models, and research that use this idea.

vLLM and SGLang use continuous batching precisely to amortize the single weight read across many concurrent requests, lifting decode throughput 5 to 10x.
FP8 KV cache and FP8 weights on H100 and B200, shipped in TensorRT-LLM, cut the bytes-per-token term that this calculation shows is the bottleneck.
Speculative decoding in production stacks serving Llama 4 and DeepSeek V4 verifies multiple draft tokens per weight read, reusing the 47 ms fetch.
Groq and Cerebras hardware chase decode latency by maximizing on-chip memory bandwidth rather than peak FLOPS, a direct bet on this ratio.
Llama 3.1 70B served on a single H100 at batch 1 shows the roughly 47 ms per-token floor predicted by this back-of-envelope estimate.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does batching change this calculation, and why does it raise throughput but not lower per-token latency?

Sign in or go Pro to see the approach for this follow-up.

QAt what context length does the ignored KV cache read start to dominate the weight read?

Sign in or go Pro to see the approach for this follow-up.

QWhy does quantizing weights to FP8 roughly halve the latency floor while quantizing activations barely helps?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.