Spot the errors in this 'optimize decode by reducing FLOPs' proposal
Click any words you think contain an error. Click again to unmark.
Single-batch decode is HBM-bandwidth-bound, not compute-bound, so cutting FLOPs by pruning or sparse heads barely helps; the real levers are quantization, KV compression, and batching.
Imagine a chef who can chop ingredients incredibly fast but works in a kitchen with one narrow doorway. Every dish needs ingredients carried in through that doorway. The chef's knife speed is not the bottleneck, the doorway is. If you make the chef chop 30 percent faster, dinner does not come out 30 percent sooner, because they still wait at the door. Decode is like this. The GPU's math units are the fast knife. The memory bus that carries the model weights and the cache into the chip is the narrow doorway. A sharper knife does not widen the doorway. To speed things up you either carry smaller ingredients each trip or cook many dinners per trip through the door.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This proposal is a single misconception wearing three disguises. Each sentence is internally plausible and each would be correct on a compute-bound workload. The trouble is that single-batch transformer decode is not compute-bound. It is memory-bandwidth-bound. Once you see that, all three claims collapse together.
The interview value here is not spotting that pruning is risky. It is demonstrating that you reach for the roofline model before you reach for any tactic. A strong candidate first classifies the workload by its bottleneck, proves the classification with a byte count, and only then proposes optimizations that attack the binding resource. A weak candidate pattern-matches FLOP reduction to latency reduction and never checks whether arithmetic was ever the scarce resource.
This deep dive establishes the roofline framing, does the concrete arithmetic for a 70B model on an H100, dismantles each of the three claims in turn, and then lays out the optimizations that actually move decode latency. By the end you should be able to sketch the roofline on a whiteboard and place this workload on it from memory.
The roofline model: two ceilings, one workload
The roofline model plots achievable throughput against arithmetic intensity, defined as FLOPs performed per byte read from memory. There are two ceilings. A sloped ceiling rises with intensity and represents the bandwidth limit. A flat ceiling represents peak compute. A workload's achievable performance is the lower of the two at its intensity.
The crossover, called the ridge point, is where the two ceilings meet. Its intensity equals peak compute divided by peak bandwidth. For an H100 that is roughly 990 TFLOP/s divided by 3.35 TB/s, about 295 FLOPs per byte. Any workload below that intensity is bandwidth-bound; any workload above it is compute-bound.
Single-batch decode has an intensity near 2 FLOPs per byte, because each weight is read once and multiplied into a single activation. That is two orders of magnitude below the ridge. Decode therefore sits deep on the bandwidth slope, nowhere near the compute roof, which is exactly why FLOP reductions do not help.
The mental picture to carry into an interview is a point sitting on the diagonal line, far to the left of the corner. To go faster you have only two moves. You can slide right by raising intensity, which lifts the achievable throughput along the slope. Or you can shrink the bytes term in the denominator, which shortens the transfer itself. Cutting FLOPs does neither, because the FLOP count never appears in the bandwidth-bound expression. That single observation is what collapses all three claims in the proposal at once.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Optimization | Cuts FLOPs? | Cuts bytes read? | Helps batch-1 decode? |
|---|---|---|---|
| Magnitude pruning (dense store) | Yes | No | Barely |
| Sparse attention heads | Yes | No | No |
| Quantization (fp8, int4) | Some | Yes | Yes |
| Batching | No | Amortizes | Yes (throughput) |
| KV compression (GQA, MLA) | Some | Yes | Yes |
Real products, models, and research that use this idea.
- vLLM and SGLang lean on continuous batching specifically because amortizing the weight read across requests is the dominant decode-throughput lever in 2026.
- DeepSeek V4 ships Multi-head Latent Attention to cut the KV bytes streamed per decode token, attacking bandwidth rather than FLOPs.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you prove decode is bandwidth-bound on your specific model and GPU?
Compute arithmetic intensity as FLOPs per byte for one decode token, then compare it to the GPU's compute-to-bandwidth ratio (the roofline ridge point). If intensity is below the ridge you are bandwidth-bound. Confirm with a profiler showing memory throughput near peak and compute utilization low.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating decode as compute-bound and chasing FLOP reductions. At batch 1 the GPU streams every weight from HBM per token, so latency tracks bytes moved, not arithmetic performed.
60 second bullets to scan on the way to the call.
Which phase is compute-bound and which is bandwidth-bound, and why
The roofline model and what arithmetic intensity means
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.