Spot the errors in this 'optimize decode by reducing FLOPs' proposal

**Decode** is the phase where the model generates output tokens one at a time. The proposal assumes this phase is compute-bound, meaning the GPU's math units are the bottleneck. That assumption is wrong, and every conclusion built on it falls apart. At batch size one, generating a single token requires reading the entire set of model weights from high-bandwidth memory into the chip, then doing a small amount of arithmetic with them. The arithmetic is fast. The reading is slow. So the GPU spends most of its time waiting for bytes to arrive, not computing. Pruning 30 percent of the feed-forward weights cuts the number of multiply-add operations by 30 percent. But if you still store the remaining weights in the same format, you move almost the same number of bytes. The math units were already idle, so making them more idle changes little. The sparse attention claim fails the same way. Skipping heads avoids some arithmetic, but the keys and values for those positions still get read from memory. You saved compute you were not short of. The real levers reduce bytes moved or amortize them. Quantization shrinks each weight. Batching reuses one weight read across many requests.

**Decode at batch one is memory-bandwidth-bound.** The proposal's root error is calling it compute-bound. Every claim it derives is therefore wrong. Reducing FLOPs does not reduce latency on a bandwidth-bound workload. The roofline model explains why. A kernel's achievable throughput is the lower of two ceilings. One ceiling is peak compute. The other is bandwidth times arithmetic intensity. Decode has tiny intensity, because each weight is read once and used for very little math. So decode sits far below the compute ceiling, pinned to the bandwidth slope. Pruning cuts FLOPs but not bytes read, so it moves you sideways along that slope. Latency barely changes. Sparse attention skips arithmetic, yet still streams the masked positions' K and V from HBM, so it does not double decode speed at batch one. Both optimizations attack a resource that was never scarce. The correct levers raise intensity or cut bytes. Quantization to int8 or fp8 cuts bytes per weight directly. Batching amortizes one weight read across many sequences, which raises intensity. KV compression with GQA or MLA cuts the cache bytes streamed per token.