Spot the errors in this 'optimize decode by reducing FLOPs' proposal · Zenaique

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Single-batch decode is HBM-bandwidth-bound, not compute-bound, so cutting FLOPs by pruning or sparse heads barely helps; the real levers are quantization, KV compression, and batching.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a chef who can chop ingredients incredibly fast but works in a kitchen with one narrow doorway. Every dish needs ingredients carried in through that doorway. The chef's knife speed is not the bottleneck, the doorway is. If you make the chef chop 30 percent faster, dinner does not come out 30 percent sooner, because they still wait at the door. Decode is like this. The GPU's math units are the fast knife. The memory bus that carries the model weights and the cache into the chip is the narrow doorway. A sharper knife does not widen the doorway. To speed things up you either carry smaller ingredients each trip or cook many dinners per trip through the door.

Key concepts

Roofline model Memory bandwidth bound Arithmetic intensity Batching

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name decode as bandwidth-bound, sketch the roofline, do the 70B-on-H100 byte math, then list the real levers (quantize, batch, KV compress) and explain why pruning and sparse heads fail.

Side-by-side

Optimization	Cuts FLOPs?	Cuts bytes read?	Helps batch-1 decode?
Magnitude pruning (dense store)	Yes	No	Barely
Sparse attention heads	Yes	No	No
Quantization (fp8, int4)	Some	Yes	Yes
Batching	No	Amortizes	Yes (throughput)
KV compression (GQA, MLA)	Some	Yes	Yes

Real-world examples

Real products, models, and research that use this idea.

vLLM and SGLang lean on continuous batching specifically because amortizing the weight read across requests is the dominant decode-throughput lever in 2026.
DeepSeek V4 ships Multi-head Latent Attention to cut the KV bytes streamed per decode token, attacking bandwidth rather than FLOPs.
NVIDIA's H100 pairs roughly 990 bf16 TFLOP/s of compute with about 3.35 TB/s of HBM, an imbalance that makes batch-one decode bandwidth-bound for any large model.
fp8 KV cache and fp8 weights in TensorRT-LLM target bytes per token directly, which is why they speed decode where pruning does not.
Speculative decoding with a draft model verifying several tokens per big-model pass raises arithmetic intensity, sidestepping the bandwidth floor.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you prove decode is bandwidth-bound on your specific model and GPU?

Sign in or go Pro to see the approach for this follow-up.

QWhy does batching shift decode from bandwidth-bound toward compute-bound?

Sign in or go Pro to see the approach for this follow-up.

QWhen does structured 2:4 sparsity actually reduce decode latency?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating decode as compute-bound and chasing FLOP reductions. At batch 1 the GPU streams every weight from HBM per token, so latency tracks bytes moved, not arithmetic performed.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.