Walk through the byte accounting that proves a single batch decode step is bandwidth bound on H100.
Use H100 SXM numbers (3 TB/s HBM, rounded from the rated 3.35 TB/s; 989 dense TFLOPS FP16) to prove that a single batch Llama-70B decode step is bandwidth bound rather than compute bound. Assume the 140 GB of FP16 weights fit on one GPU (in practice they need at least two 80 GB H100s). Show the byte accounting and the time each phase takes.
At batch 1 each decode step reads all 140 GB of FP16 weights once for tiny GEMV compute, so latency tracks bytes over bandwidth, roughly 47 ms on H100.
Picture a chef who can chop in a blink but keeps every recipe in a giant cookbook locked in a back room. To cook one dish the chef must walk to the back room, haul out the whole cookbook, read a single line, then cook. The walking and carrying takes minutes. The cooking takes a second. The bottleneck is the trip, not the cooking. A decode step is the same. The GPU must drag every model weight out of memory just to produce one token. The actual arithmetic is trivial, so the time is set entirely by how fast the memory bus moves the weights. Cooking for one customer or twenty takes the same single trip, which is why serving many requests together is so much cheaper per dish.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Two divisions show that decode is bandwidth bound on H100. The weight read takes about 47 ms while the arithmetic takes about 0.14 ms. So memory bandwidth dominates compute by more than two orders of magnitude. The single batch decode step is the standard example of a memory-bound workload in machine learning systems.
The conclusion is the opposite of what many engineers expect from training. Training is compute bound and rewards faster tensor cores, bigger matmuls, and better FLOP utilization. Single-stream decode is the opposite. The tensor cores idle while the weights stream in. The headline metric is how many bytes you can move per token, not how many FLOPs you can issue per second. So every decode optimization that lowers latency must reduce one term: the bytes moved per token.
Why a decode step reads every weight once
Autoregressive decode emits one token at a time. To produce the next token the model runs a full forward pass over a single new position. So every weight matrix in every layer must be applied to exactly one activation vector. There is no way around this: the next token depends on the output of every layer, so every weight participates.
Applying a weight matrix to a single vector is a matrix-vector product, a GEMV. Unlike the matrix-matrix products of prefill or training, a GEMV touches each weight exactly once and performs a single multiply-add with it before moving on. There is no reuse to amortize, no inner tile that gets read many times, nothing the cache hierarchy can hide. The moment a weight is loaded it is used once and discarded.
This is the main cause of the bottleneck. The arithmetic intensity, defined as FLOPs performed per byte loaded, is close to 1 for a GEMV. Hardware needs intensities in the hundreds to keep its tensor cores busy, so a GEMV leaves the compute units almost entirely idle. The step time is then set by how long it takes to read the weights from HBM, not by how long the math takes. This is why the same model that is comfortably compute bound during training becomes severely memory bound during single-stream decode. The only thing that changed is that the batch dimension collapsed to 1, which turns every GEMM into a GEMV.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Lever | Effect on bytes | New floor (70B) | Composes with |
|---|---|---|---|
| FP16 baseline | 140 GB read | 47 ms | baseline |
| FP8 quant | 70 GB read | 23 ms | Batching, TP |
| INT4 quant | 35 GB read | 11.7 ms | Batching, TP |
| Batch B=64 | Same 140 GB per step, 64 tokens out | 47 ms for 64 tokens | Quant, TP |
| TP-8 | 140 GB across 8 HBMs | 5.8 ms plus allreduce | Quant, Batching |
Real products, models, and research that use this idea.
- A 70B FP16 model does not fit on one 80 GB H100, so it is served across at least two. With two-way tensor parallelism each GPU streams 70 GB per step. That halves the batch 1 floor to about 23 ms and caps decode near 43 tokens per second per request. Measured rates sit below that ceiling once allreduce and kernel overhead are added.
- Serving stacks such as vLLM, SGLang and TensorRT LLM combine weight quantization, continuous batching and tensor parallelism, the three levers that move the decode floor.
What an interviewer would ask next. Try answering before peeking at the approach.
QRecompute the floor for 7B FP16, 70B INT4, and 70B FP16 with 8-way tensor parallelism.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quoting decode cost in FLOPs and reaching for faster matmul kernels. Decode at batch 1 is memory bound, so tensor core throughput is irrelevant. Only bytes moved and bandwidth set the floor.
60 second bullets to scan on the way to the call.
Why decode at batch 1 reads the entire weight matrix once per token
How to compute the FP16 weight byte count for a 70B model
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.