Explain why a textbook ML optimization instinct (reduce FLOPs to speed up inference) fails on LLM decode. What does work, and what was the engineer's confusion?
Free · 2 AI evals / day
Most engineers learn optimization in regimes where speed tracks **FLOPs**, the count of floating-point operations. Training and image-model inference push large batches through the GPU, so the chip's math units are the bottleneck. Cut the math and you finish faster. That instinct is correct there.
LLM decode breaks the instinct. Generating one token at a time, with one or few requests in flight, means the GPU does very little math per step. The real cost is reading the model's weights out of high-bandwidth memory so the math units have something to multiply.
For a 70B model in FP16 that read is roughly 140 GB every single token. On an H100 with about 3.3 TB per second of bandwidth, that read alone takes tens of milliseconds, and the arithmetic finishes long before the bytes arrive.
So pruning, sparse activations, or clever attention that cut FLOPs leave that memory read untouched. The token latency barely changes, because you optimized the part that was never the bottleneck.
What actually helps is cutting bytes moved or moving them once for many tokens: quantizing weights, shrinking the KV cache with GQA, batching requests, and speculative decoding.
**The misconception** is treating FLOPs as the universal speed lever. It comes from training and CNN inference, which are compute-bound. There, cutting math directly cuts time.
LLM decode at small batch is different. It sits on the bandwidth slope of the roofline, not the compute roof. Every decode step must stream the full weight matrix from HBM to produce just one token. A 70B model in FP16 is about 140 GB. On an H100 near 3.3 TB per second, that read costs tens of milliseconds per token, and the matmuls finish well before the bytes land.
So a GPU with more peak FLOPs but the same HBM bandwidth barely moves token latency. The arithmetic was already cheap relative to the read. Pruning, sparse activations, and sparse attention cut FLOPs but not bytes, so they rarely help decode.
The levers that work all reduce bytes or amortize the read. Weight quantization to INT4 quarters the HBM traffic. GQA, MLA, and KV quantization shrink the per-token cache read. Batching amortizes one weight read across many requests. Speculative decoding verifies several tokens per weight read.
The diagnostic rule is to identify which side of the roofline you are on first, then pick the matching lever.
Quick reference
TL;DR
Decode at small batch is HBM-bandwidth-bound, not compute-bound. Cutting FLOPs without cutting bytes-per-token leaves the dominant memory read unchanged, so latency barely moves.
Imagine a chef who must walk to a giant pantry and carry back every ingredient in the kitchen before cooking each single dish. The walk takes a long time; the actual chopping is quick. If you give the chef faster knives, dinner barely speeds up, because the walk is the bottleneck, not the chopping. LLM decode is like that. Generating one token forces the GPU to read the entire model from slow memory, which dwarfs the tiny amount of math per token. Buying a chip with more raw math power, like sharper knives, helps almost nothing if the pantry walk, the memory bandwidth, stays the same. To go faster you shrink what must be carried or carry it once for many dishes.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Reasoning about prefill latency on long input prompts Prefill is compute-bound because one weight read amortizes over many parallel tokens, so FLOP-side analysis applies there.
Very large batch decode that has crossed the roofline ridge Once intensity passes the ridge the workload is compute-bound, and reducing FLOPs or using faster math units genuinely helps.
Tiny models that fit entirely in on-chip SRAM or cache When weights do not stream from HBM each step, the bandwidth argument weakens and compute can dominate again.
Workloads dominated by tokenization, networking, or scheduling overhead If the bottleneck is outside the GPU forward pass, neither FLOP nor bandwidth tuning of the kernel will help much.
Quick scan · 2–4 min
Everything important, quickly.
4 min: roofline regimes + where decode sits + estimate the per-token weight read + why FLOP cuts miss + byte-cutting and amortization levers + the diagnostic rule.
Real-world examples
Real products, models, and research that use this idea.
1vLLM with continuous batching raises arithmetic intensity by sharing one weight read across many requests, the largest decode throughput lever in 2026 serving.
2Llama 3.1 70B served on H100 spends most of each decode step on the FP16 weight read, which dwarfs the matmul time.
3DeepSeek V4 ships Multi-head Latent Attention to shrink the per-token KV bandwidth bill rather than chasing FLOP cuts.
4INT4 weight quantization via AWQ or GPTQ roughly quarters the HBM read and is a standard decode-latency win on production endpoints.
5NVIDIA B200 lifts decode throughput mainly by raising HBM bandwidth, not just peak FLOPs, confirming bandwidth sets small batch decode speed.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QAt what batch size does decode cross from bandwidth-bound to compute-bound?
Sign in or go Pro to see the approach for this follow-up.
QWhy does prefill behave compute-bound while decode is bandwidth-bound on the same model?
Sign in or go Pro to see the approach for this follow-up.
QDoes speculative decoding reduce FLOPs or just rearrange the bandwidth bill?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming peak FLOPs sets decode speed. At batch 1 decode is gated by reading weights from HBM, so a chip with more FLOPs but equal bandwidth barely helps token latency.