TPOT is high, so the team plans to upgrade A100 to a faster compute GPU, critique

**Decode time per token has two parts.** First, the GPU reads the model weights from main memory (HBM) into the chip. Second, it does the math on those weights to produce the next token. The reading part is much slower than the math part, so it sets the floor on how fast tokens can come out. For a 70B model in bf16, the weights are about 140 gigabytes. An A100's HBM bandwidth is 2 terabytes per second. Dividing those two numbers gives you about 70 milliseconds just to read the weights once. That is the minimum time per token on this GPU, regardless of how fast the math is. The team's proposal doubles the math speed but keeps the HBM bandwidth at 2 TB/s. The math was already fast enough to fit in the 70-millisecond window. Making it twice as fast does almost nothing, because the slow part, the memory read, is unchanged. Realized TPOT might drop a few milliseconds at most. The right move is to attack the slow part. Switch to a GPU with more HBM bandwidth. An H100 has about 3.3 TB/s, so the 140-gigabyte read takes about 42 milliseconds. An H200 has about 4.8 TB/s, so the same read takes about 29 milliseconds. A B200 at about 8 TB/s gets it down to about 18 milliseconds. Each step of bandwidth upgrade really does cut TPOT. The other option is to shrink the bytes themselves: FP8 weights are half the size of bf16, so the read halves.

**The team is buying compute they cannot use.** Decode at batch 1 on a 70B model is firmly memory-bandwidth-bound. Per-step time is dominated by streaming the full weight set out of HBM, which is approximately 140 GB at bf16. On an A100 at 2 TB/s, that read takes about 70 ms; the matmul fits inside that window. Doubling peak FP16 TFLOPs while holding HBM bandwidth at 2 TB/s shrinks an already-overlapped subproblem and leaves the 70 ms floor intact. The predicted 50 percent TPOT improvement is unsupported by the math. The right axis is HBM bandwidth. The candidate GPUs and their TPOT floors for a 70B bf16 model are: A100 at 2 TB/s gives ~70 ms; H100 at 3.3 TB/s gives ~42 ms; H200 at 4.8 TB/s gives ~29 ms; B200 at 8 TB/s gives ~18 ms. The SLA of 50 ms is met by H100 or better. The proposed 'doubled FLOPs, same bandwidth' SKU does not meet it. The other equivalent lever is to reduce bytes-moved-per-step rather than to raise bandwidth. FP8 weight quantization halves the 140 GB read to 70 GB, cutting TPOT roughly in half on the same hardware; on A100 that gives ~35 ms, meeting SLA. INT4 quarters it. GQA-8 or MLA shrink the KV term of the read, which matters more at long contexts. Note what is missing from the menu: 'more FLOPs.' Compute is not the bottleneck at batch 1 on a 70B model on any current production GPU. Adding TFLOPs only matters when the workload crosses the bandwidth-to-compute knee, which for this scenario sits somewhere around batch 128-256 depending on the exact hardware. The team is operating well below that point and would not feel the extra compute. The diagnostic to apply in any decode-TPOT review: compute the bytes-per-step (weights plus KV cache), divide by candidate GPU bandwidth, and confirm whether the resulting floor meets SLA. If it does not, the upgrade must move bandwidth or move bytes, not FLOPs.