Walk through the byte accounting that proves a single batch decode step is bandwidth bound on H100.
Use H100 numbers (3 TB/s HBM, 989 TFLOPS FP16) to prove that a single batch Llama-70B decode step is bandwidth bound rather than compute bound. Show the byte accounting and the time each phase takes.
At batch 1 each decode step reads all 140 GB of FP16 weights once for tiny GEMV compute, so latency tracks bytes over bandwidth, roughly 47 ms on H100.
Picture a chef who can chop in a blink but keeps every recipe in a giant cookbook locked in a back room. To cook one dish the chef must walk to the back room, haul out the whole cookbook, read a single line, then cook. The walking and carrying takes minutes; the cooking takes a second. The bottleneck is the trip, not the cooking. A decode step is the same. The GPU must drag every model weight out of memory just to produce one token. The actual arithmetic is trivial, so the time is set entirely by how fast the memory bus moves the weights. Cooking for one customer or twenty takes the same single trip, which is why serving many requests together is so much cheaper per dish.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Decode being bandwidth bound on H100 is not a hand-wave; it falls out of two divisions, and the result is unambiguous. The weight read takes about 47 ms while the arithmetic takes about 0.14 ms, so memory bandwidth dominates compute by more than two orders of magnitude. The single batch decode step is the canonical memory-bound workload in all of machine learning systems, and it is one of the most reliable hard interview questions in inference optimization precisely because it separates candidates who have read about serving from those who have actually stared at a decode trace.
This matters because the conclusion inverts the intuition many candidates bring from training. Training is compute bound and rewards faster tensor cores, bigger matmuls, and better FLOP utilization. Single-stream decode is the opposite: the tensor cores idle while the weights stream in, and the headline metric is how many bytes you can move per token, not how many FLOPs you can issue per second. Once you internalize that, the entire decode-side optimization curriculum reorganizes itself around one term, the bytes you must move per token, and every serious lever turns out to be an attack on that single term.
This deep dive does the accounting step by step. It states the governing relation, computes both times with the H100 numbers the prompt provides, interprets the ratio through the roofline model, and then derives the three levers that actually move the floor. By the end you should be able to redo the arithmetic on a whiteboard, defend each number, and explain why a faster matmul kernel changes nothing about single-stream decode latency.
Why a decode step reads every weight once
Autoregressive decode emits one token at a time. To produce the next token the model runs a full forward pass over a single new position, which means every weight matrix in every layer must be applied to exactly one activation vector. There is no way around this: the next token depends on the output of every layer, so every weight participates.
Applying a weight matrix to a single vector is a matrix-vector product, a GEMV. Unlike the matrix-matrix products of prefill or training, a GEMV touches each weight exactly once and performs a single multiply-add with it before moving on. There is no reuse to amortize, no inner tile that gets read many times, nothing the cache hierarchy can hide. The moment a weight is loaded it is consumed and discarded.
That is the heart of the problem. The arithmetic intensity, defined as FLOPs performed per byte loaded, is close to 1 for a GEMV. Hardware needs intensities in the hundreds to keep its tensor cores busy, so a GEMV leaves the compute units almost entirely idle. The step time is then set by how long it takes to read the weights from HBM, not by how long the math takes. This is why the same model that is comfortably compute bound during training flips to severely memory bound during single-stream decode: the only thing that changed is that the batch dimension collapsed to 1, turning every GEMM into a GEMV.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Lever | Effect on bytes | New floor (70B) | Composes with |
|---|---|---|---|
| FP16 baseline | 140 GB read | 47 ms | baseline |
| FP8 quant | 70 GB read | 23 ms | Batching, TP |
| INT4 quant | 35 GB read | 11.7 ms | Batching, TP |
| Batch B=64 | Same 140 GB per step, 64 tokens out | 47 ms for 64 tokens | Quant, TP |
| TP-8 | 140 GB across 8 HBMs | 5.8 ms plus allreduce | Quant, Batching |
Real products, models, and research that use this idea.
- vLLM batch 1 Llama 3.1 70B at FP16 measures near 21 tokens per second per request, matching the roughly 47 ms per-step floor this math predicts.
- Together AI serves Llama 3.1 70B with INT4 weights plus 8-way tensor parallelism to push past 100 tokens per second per request.
What an interviewer would ask next. Try answering before peeking at the approach.
QRecompute the floor for 7B FP16, 70B INT4, and 70B FP16 with 8-way tensor parallelism.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quoting decode cost in FLOPs and reaching for faster matmul kernels. Decode at batch 1 is memory bound, so tensor core throughput is irrelevant; only bytes moved and bandwidth set the floor.
60 second bullets to scan on the way to the call.
Why decode at batch 1 reads the entire weight matrix once per token
How to compute the FP16 weight byte count for a 70B model
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.