Find the wrong move in 'decode is slow, so let's switch to a smaller FLOP model'

Spot the error·Medium·4.0 · 0·~2 min·Asked atCred Jasper·Relevant atNVIDIA

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

**Decode** in a large language model produces one token at a time. For each new token the GPU must read the model's weights and the growing key-value cache from its main memory (HBM), do a small amount of math, and write the result back. The slow part is reading the data, not doing the math. This is what people mean by memory-bandwidth-bound. At batch size one, the arithmetic per token is a thin matrix-vector multiplication. Modern GPUs finish that math much faster than they can deliver the weights it needs. So while the math units are idle, the memory bus is pegged. The plan in the question switches to a model with thirty percent fewer attention FLOPs but the same weight count and the same KV cache size. The bytes that get streamed from HBM on each token step are unchanged. The math gets faster, but the math was never the limit. The realized TPOT improvement is close to zero, not the predicted thirty percent. The correct levers attack bytes-per-token. Quantize the weights from bf16 to FP8 or INT4 to halve or quarter the weight read. Switch to a model with GQA or MLA to shrink the KV cache. Batch many requests together so each HBM read serves many tokens. Less math, same memory, does nothing.

**The proposal optimizes the wrong axis.** At batch 1, decode is firmly memory-bandwidth-bound. Each step streams the full weight set plus the KV cache from HBM; the per-token math is a thin matrix-vector product the compute units finish in a fraction of the bandwidth time. Cutting FLOPs by thirty percent while leaving bytes-per-token unchanged shaves time off a subproblem that was already overlapped with the memory read, so net wall-clock improvement is closer to zero than thirty. Formally, the roofline at batch 1 sits on the bandwidth slope. Per-step time is approximately bytes-read divided by HBM bandwidth, capped from below by that ratio regardless of FLOP rate. For a 70B bf16 model the weight read alone is about 140 gigabytes per step; on an A100 at 2 TB/s that is roughly 70 ms of unavoidable memory time. The matmul fits inside that window with room to spare. Removing some of the matmul does not shrink the window. The correct levers all target bytes-moved-per-step. FP8 weights halve the weight read to roughly 70 GB, cutting TPOT roughly in half on the same hardware. INT4 quartiles it. GQA-8 or MLA shrink the KV cache by eight to one hundred times, easing the KV term of the read. Batching amortizes the weight read across many requests, raising tokens per second per GPU dramatically while leaving any one request's latency roughly fixed. Moving to a higher-bandwidth GPU (H100 at 3.3 TB/s, H200 at 4.8, B200 at 8) cuts the read time directly. The FLOP-reduction proposal would only matter if the workload were prefill, which is compute-bound, or if the batch were large enough to push decode past the compute crossover. Neither is the case here, so the prediction of thirty percent TPOT improvement is unsupported by the math.

TL;DR

Decode is memory-bandwidth-bound at batch 1: per-step time is set by reading weights and KV cache from HBM, not by the matmul.

Memory aid

Easy to grasp

Picture a chef whose kitchen is so small that they can only carry one armful of ingredients at a time from the pantry. Once the ingredients are on the counter, chopping them is fast; the slow part is the trip to the pantry and back. Speeding up the chopping does nothing if the chef still has to make the same long walk for the same ingredients. Decode in a large language model is like that. Each new token forces the GPU to fetch the same enormous bag of weights from far-away memory. The actual math is done in a blink once the data arrives. A model that does thirty percent less chopping still walks the same distance to fetch the same load, so the meal is not faster.

Key concepts

Roofline model Memory-bandwidth-bound Decode phase Time per output token

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The proposal in front of us is the most common wrong move in LLM inference optimization: assume decode is compute-bound and shop for a model with cheaper math. The instinct comes from training, where compute really does dominate, and from intro performance courses, where FLOPs are the headline metric. Inference decode does not work that way, and the difference is not a small correction but a complete inversion of which axis to optimize.

The author predicts that swapping in a variant with thirty percent fewer attention FLOPs will cut decode TPOT by thirty percent. To see why that prediction is wrong, we need to be precise about what happens during a single decode step on a 70B model, what is actually saturated, and what consequences follow for the menu of optimizations the team should be running instead.

The goal of this walkthrough is to give you the roofline mental model in a form you can apply to any decode workload in under a minute: estimate the bytes moved per step, divide by the GPU's HBM bandwidth to get a TPOT floor, and check whether the workload is sitting on the bandwidth slope or under the compute ceiling. Once you carry that picture in your head, the FLOP-reduction proposal classifies itself as a wrong-axis move and the right levers fall out naturally.

Anatomy of a single decode step

At batch size one, every decode step does roughly the following. The GPU reads the full set of model weights out of HBM into on-chip memory layer by layer. It also reads the KV cache entries accumulated for every prior token. With weights and KV in the streaming-multiprocessor caches, it performs a matrix-vector multiplication for each layer: a vector of size hidden-dim against a weight matrix of size hidden-dim by hidden-dim, plus the attention work over the cached keys and values. It writes a small activation back to HBM, samples the next token, and appends one entry per layer to the KV cache for use on the next step.

The central observation is the asymmetry between the bytes read and the FLOPs done. For a 70B model in bf16, the weight set alone is around 140 gigabytes. At a context of a few thousand tokens, the KV cache adds another tens of gigabytes. So the per-step memory traffic is well over a hundred gigabytes. The per-step FLOPs, by contrast, are a few hundred gigaflops, because each loaded weight gets multiplied by a single vector slice and then discarded. Arithmetic intensity, the ratio of FLOPs to bytes, is in the single digits.

GPUs are designed for arithmetic intensities in the hundreds. An H100 has roughly 200 FLOPs per byte of HBM bandwidth at peak; the A100 is similar. So a workload running at intensity five sits orders of magnitude below the hardware balance. The tensor cores spend most of their time idle, waiting for the next slice of weights to arrive from HBM. The wall-clock per step is set by the memory traffic, not the math.

The roofline picture and the TPOT floor

The right menu of levers

When the FLOP argument actually applies

What to say in the design review

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM and SGLang both report that single-stream Llama-3 70B decode at batch 1 sustains roughly 30-50 tok/s on A100 and 70-100 tok/s on H100, scaling with HBM bandwidth rather than FLOPs.
NVIDIA's H200 release notes pitch the 76% bandwidth bump over H100 as the headline inference win, not the unchanged FLOP rate.

What an interviewer would ask next. Try answering before peeking at the approach.

QAt what batch size does decode cross over from bandwidth-bound to compute-bound on a 70B model on H100?

Compute the crossover from arithmetic intensity. H100 has roughly 200 FLOPs/byte balance. Decode at batch B has arithmetic intensity scaling linearly with B because each loaded weight gets reused across B requests. Solve for B where intensity hits 200. For 70B bf16 on H100 this lands around batch 128-256; below that you are bandwidth-bound, above it you are compute-bound.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating decode as compute-bound and shopping for lower-FLOP models. At batch 1, decode is bandwidth-bound; the matmul finishes well before the HBM read, so cheaper math saves nothing on wall-clock.

60 second bullets to scan on the way to the call.

Why decode is bandwidth-bound at batch 1 while prefill is compute-bound
How to derive TPOT floor from weight size and HBM bandwidth

Primary sources. Browse if you want the original framing.