Click any words you think contain an error. Click again to unmark.
Decode is bandwidth-bound, so a GPU upgrade that doubles FLOPs while keeping HBM bandwidth flat barely changes TPOT. Per-step time floor = weight bytes / HBM bandwidth (~70ms for 70B bf16 on A100).
Picture a water hose filling a bucket. You can buy a much bigger bucket, but if the hose is the same size, the bucket fills at the same rate. The hose is HBM bandwidth, and the bucket is compute. Decode keeps asking for the same large amount of data on every token. Doubling the bucket size does not make the water arrive faster. To speed things up you need a bigger hose, like the H100 or B200, or you need to ask for less water, like storing the model's weights in 8 bits instead of 16.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Choosing a GPU by peak TFLOPs to fix decode TPOT is a common and costly mistake. TPOT is too high, and budget goes to a hardware upgrade. The team compares candidates on peak TFLOPs and picks the largest compute number. A short calculation shows the mistake.
The proposal: move from A100 (2 TB/s HBM, 312 dense BF16 TFLOPS) to an unnamed SKU with double the FLOPs but the same 2 TB/s of HBM bandwidth, and expect TPOT to drop from 80 ms to 40 ms. The prediction assumes the matmul is the slow part of a decode step. It is not. On a 70B bf16 model at batch 1, the matmul takes a small fraction of the time spent moving weights from HBM. So halving matmul time saves a small fraction of TPOT.
A bytes-reduction path (8-bit or 4-bit weights, GQA, MLA) can meet the SLA without new hardware. The same floor formula and the candidates' bandwidth figures can test any GPU proposal for decode TPOT.
The TPOT floor formula
A single decode step on a transformer LLM at batch 1 does roughly this. For each layer, read the layer's weight tensors from HBM into the streaming-multiprocessor caches, and read the layer's KV cache entries. Then perform a matrix-vector multiplication for the QKV projections plus attention plus the MLP, and write a small activation back. The cycle repeats per layer, then the sampler produces the next token, then a new KV entry is appended per layer.
The time per step is the sum of memory time and compute time, with overlap. In the bandwidth-bound regime (decode at batch 1 on a large model), memory time dominates and sets the floor. Memory time is bytes-read divided by HBM bandwidth. Bytes-read is the sum of weight bytes plus KV-cache bytes plus a small term for activations.
For a 70B model in bf16, weight bytes are about 70 billion parameters x 2 bytes = 140 GB. Llama 3.1 70B has 80 layers and 8 KV heads of dimension 128, so its KV cache at 4k tokens is about 1.3 GB. Per-step memory traffic is therefore about 141 GB. Meeting the 50 ms SLA needs about 141 GB / 0.05 s = 2.8 TB/s, so 2 TB/s falls short by about 1.4x. These figures assume one GPU's bandwidth carries the whole read. A 140 GB model needs at least two 80 GB GPUs. With N-way tensor parallelism each GPU reads 1/N of the bytes, so divide every floor by N. The ratios between options stay the same.
The compute time on A100 at 312 dense BF16 TFLOPS: per-step FLOPs (about 2 per parameter, so about 140 GFLOPs) / peak gives 140e9 / 312e12 = 0.45 ms. That is less than 1 ms against a memory time of 70 ms, so the compute units are idle for about 99 percent of each step.
Now the proposal: double the compute. Compute time drops to about 0.22 ms. Memory time is still 70 ms, so per-step time is still about 70 ms. The upgrade saves at most about 0.2 ms of an 80 ms step, and TPOT moves from 80 ms to about 79.8 ms. The predicted 40 ms saving is about 200 times the real one.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- NVIDIA pitched the H200 mainly on 1.4x HBM bandwidth (4.8 TB/s against H100's 3.35) for inference, with FLOPs unchanged from H100.
- B200 raises bandwidth to 8 TB/s and adds FP4 Tensor Cores. FP4 weights on B200 give TPOT floors below 10 ms even on 70B models.
What an interviewer would ask next. Try answering before peeking at the approach.
QAt what batch size does a 70B model on H100 cross from bandwidth-bound to compute-bound during decode?
Compute arithmetic intensity at batch B: each loaded weight is reused across B requests, so the intensity of the bf16 weight matmuls is about B FLOPs per byte. H100 SXM balances at about 295 FLOPs per byte (989 dense BF16 TFLOPS / 3.35 TB/s). At batch 1 the intensity is about 1, so the crossover is near batch 300. Each sequence also reads its own KV cache, which stays memory-bound, so long contexts push the crossover higher. Past it, FLOPs matter.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Buying more compute to fix a decode TPOT problem. Decode is bandwidth-bound at batch 1, so additional TFLOPs sit idle waiting for HBM. The right upgrade adds bandwidth (H100/H200/B200) or shrinks bytes per step (FP8, GQA, MLA).
60 second bullets to scan on the way to the call.
How to compute the per-step TPOT floor from weight bytes and HBM bandwidth
HBM bandwidth specs for A100, H100, H200, B200
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.