Why is 'reduce FLOPs to speed up decode' the most common beginner misconception in LLM serving?

Most engineers learn optimization in regimes where speed tracks **FLOPs**, the count of floating-point operations. Training and image-model inference push large batches through the GPU, so the chip's math units are the bottleneck. Cut the math and you finish faster. That instinct is correct there. LLM decode breaks the instinct. Generating one token at a time, with one or few requests in flight, means the GPU does very little math per step. The real cost is reading the model's weights out of high-bandwidth memory so the math units have something to multiply. For a 70B model in FP16 that read is roughly 140 GB every single token. On an H100 with about 3.3 TB per second of bandwidth, that read alone takes tens of milliseconds, and the arithmetic finishes long before the bytes arrive. So pruning, sparse activations, or clever attention that cut FLOPs leave that memory read untouched. The token latency barely changes, because you optimized the part that was never the bottleneck. What actually helps is cutting bytes moved or moving them once for many tokens: quantizing weights, shrinking the KV cache with GQA, batching requests, and speculative decoding.

**The misconception** is treating FLOPs as the universal speed lever. It comes from training and CNN inference, which are compute-bound. There, cutting math directly cuts time. LLM decode at small batch is different. It sits on the bandwidth slope of the roofline, not the compute roof. Every decode step must stream the full weight matrix from HBM to produce just one token. A 70B model in FP16 is about 140 GB. On an H100 near 3.3 TB per second, that read costs tens of milliseconds per token, and the matmuls finish well before the bytes land. So a GPU with more peak FLOPs but the same HBM bandwidth barely moves token latency. The arithmetic was already cheap relative to the read. Pruning, sparse activations, and sparse attention cut FLOPs but not bytes, so they rarely help decode. The levers that work all reduce bytes or amortize the read. Weight quantization to INT4 quarters the HBM traffic. GQA, MLA, and KV quantization shrink the per-token cache read. Batching amortizes one weight read across many requests. Speculative decoding verifies several tokens per weight read. The diagnostic rule is to identify which side of the roofline you are on first, then pick the matching lever.