Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
**Decode time per token has two parts.** First, the GPU reads the model weights from main memory (HBM) into the chip. Second, it does the math on those weights to produce the next token. The reading part is much slower than the math part, so it sets the floor on how fast tokens can come out.
For a 70B model in bf16, the weights are about 140 gigabytes. An A100's HBM bandwidth is 2 terabytes per second. Dividing those two numbers gives you about 70 milliseconds just to read the weights once. That is the minimum time per token on this GPU, regardless of how fast the math is.
The team's proposal doubles the math speed but keeps the HBM bandwidth at 2 TB/s. The math was already fast enough to fit in the 70-millisecond window. Making it twice as fast does almost nothing, because the slow part, the memory read, is unchanged. Realized TPOT might drop a few milliseconds at most.
The right move is to attack the slow part. Switch to a GPU with more HBM bandwidth. An H100 has about 3.3 TB/s, so the 140-gigabyte read takes about 42 milliseconds. An H200 has about 4.8 TB/s, so the same read takes about 29 milliseconds. A B200 at about 8 TB/s gets it down to about 18 milliseconds. Each step of bandwidth upgrade really does cut TPOT. The other option is to shrink the bytes themselves: FP8 weights are half the size of bf16, so the read halves.
**The team is buying compute they cannot use.** Decode at batch 1 on a 70B model is firmly memory-bandwidth-bound. Per-step time is dominated by streaming the full weight set out of HBM, which is approximately 140 GB at bf16. On an A100 at 2 TB/s, that read takes about 70 ms; the matmul fits inside that window. Doubling peak FP16 TFLOPs while holding HBM bandwidth at 2 TB/s shrinks an already-overlapped subproblem and leaves the 70 ms floor intact. The predicted 50 percent TPOT improvement is unsupported by the math.
The right axis is HBM bandwidth. The candidate GPUs and their TPOT floors for a 70B bf16 model are: A100 at 2 TB/s gives ~70 ms; H100 at 3.3 TB/s gives ~42 ms; H200 at 4.8 TB/s gives ~29 ms; B200 at 8 TB/s gives ~18 ms. The SLA of 50 ms is met by H100 or better. The proposed 'doubled FLOPs, same bandwidth' SKU does not meet it.
The other equivalent lever is to reduce bytes-moved-per-step rather than to raise bandwidth. FP8 weight quantization halves the 140 GB read to 70 GB, cutting TPOT roughly in half on the same hardware; on A100 that gives ~35 ms, meeting SLA. INT4 quarters it. GQA-8 or MLA shrink the KV term of the read, which matters more at long contexts.
Note what is missing from the menu: 'more FLOPs.' Compute is not the bottleneck at batch 1 on a 70B model on any current production GPU. Adding TFLOPs only matters when the workload crosses the bandwidth-to-compute knee, which for this scenario sits somewhere around batch 128-256 depending on the exact hardware. The team is operating well below that point and would not feel the extra compute.
The diagnostic to apply in any decode-TPOT review: compute the bytes-per-step (weights plus KV cache), divide by candidate GPU bandwidth, and confirm whether the resulting floor meets SLA. If it does not, the upgrade must move bandwidth or move bytes, not FLOPs.
Quick reference
TL;DR
Decode is bandwidth-bound, so a GPU upgrade that doubles FLOPs while keeping HBM bandwidth flat barely changes TPOT. Per-step time floor = weight bytes / HBM bandwidth (~70ms for 70B bf16 on A100).
Picture a water hose filling a bucket. You can buy a much bigger bucket, but if the hose is the same size, the bucket fills at the same rate. The hose is HBM bandwidth, the bucket is compute. Decode keeps demanding the same big drink of data on every token; doubling the bucket size does not make the water arrive faster. To speed things up you need a bigger hose, like the H100 or B200, or you need to ask for less water, like switching to a smaller-bytes model with FP8 weights.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Prefill latency on long prompts (TTFT) Prefill is compute-bound on long prompts because each loaded weight is matmul'd against the prompt matrix, raising arithmetic intensity. FLOP increases do help prefill, so the FLOP-doubling SKU might be the right tool for a TTFT-bound workload.
Very large batch sizes past the bandwidth to compute crossover At batch 128+ on H100, a 70B workload pushes against the compute ceiling rather than the bandwidth slope, and additional FLOPs start to matter.
Small models (sub-7B) on top-tier GPUs Small weight sets reduce the bandwidth read, so compute can become binding even at modest batch sizes; FLOPs matter for these workloads.
Training and fine-tuning workloads Training is dominated by matrix-matrix multiplications with high arithmetic intensity, where peak FLOPs directly determine throughput.
Quick scan · 2–4 min
Everything important, quickly.
3 min: compute the A100 TPOT floor (140 GB / 2 TB/s = 70 ms), show that doubling FLOPs at same bandwidth does not move it, list bandwidth-upgrade options with their floors, present FP8 as the zero-hardware alternative.
Real-world examples
Real products, models, and research that use this idea.
1NVIDIA H200 release in late 2024 was pitched primarily on 1.4x HBM bandwidth (4.8 TB/s vs H100's 3.35) for inference, with FLOPs unchanged from H100.
2B200 in 2025 doubled bandwidth again to ~8 TB/s with FP4 tensor cores; FP4 weight reads on B200 deliver TPOT floors below 10ms even on 70B models.
3vLLM benchmarks on Llama-3 70B show single-stream TPOT scaling near-linearly with HBM bandwidth across A100/H100/H200, confirming the bandwidth-floor model.
4TensorRT-LLM FP8 path on H100 routinely delivers ~1.7-1.9x TPOT improvement for 70B models, matching the predicted bandwidth-bound scaling from halving bytes-per-step.
5Groq's LPU achieves >1000 tok/s single-stream on smaller models by pairing modest compute with extreme on-chip SRAM bandwidth and no HBM read at all; a direct empirical confirmation that bandwidth, not FLOPs, drives decode latency.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QAt what batch size does a 70B model on H100 cross from bandwidth-bound to compute-bound during decode?
Sign in or go Pro to see the approach for this follow-up.
QHow does the KV cache contribute to the per-step bytes read, and when does it dominate?
Sign in or go Pro to see the approach for this follow-up.
QWhy has NVIDIA's roadmap grown HBM bandwidth faster than peak FLOPs in recent generations?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Buying more compute to fix a decode TPOT problem. Decode is bandwidth-bound at batch 1, so additional TFLOPs sit idle waiting for HBM. The right upgrade adds bandwidth (H100/H200/B200) or shrinks bytes per step (FP8, GQA, MLA).