Derive the arithmetic intensity (FLOPs per byte read from HBM) of one decode step at batch 1 and at batch B, ignoring the KV cache contribution. Use a model with N parameters in FP16. Explain what the result implies about decode's place on the roofline.
At batch 1 a decode step reads 2N weight bytes and does 2N FLOPs, so arithmetic intensity is 1 FLOP per byte. Batching to B raises it to B, sliding decode up the roofline toward compute-bound.
Imagine a chef who must walk to a giant pantry, carry back every ingredient, then cook one tiny dish. The walk is the slow part, not the cooking. Serving one customer, almost all the time is spent fetching, so the stove sits nearly idle. Now imagine forty customers ordering at once. The chef makes one trip to the pantry and cooks forty dishes from the same haul. The walk cost is shared, so the stove finally runs hot. LLM decode is the same. Reading the model weights from memory is the long walk; the math is the quick cooking. One request wastes the hardware; batching many requests pays for the trip once and keeps the chips busy.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Arithmetic intensity is the single number that explains why LLM decode is slow, why batching is the dominant throughput lever, and why "more FLOPs" rarely fixes a serving system. It is defined as the ratio of compute operations to bytes moved between the GPU's compute cores and its high-bandwidth memory.
The roofline model uses this number to classify any kernel. If a kernel does little math per byte, it spends its time waiting for memory and is called memory-bound or bandwidth-bound. If it does lots of math per byte, the compute units are the limit and it is compute-bound. The dividing line is the critical intensity of the hardware, the x-coordinate of the roofline ridge.
Decode at batch 1 is the textbook memory-bound kernel. This deep dive derives its intensity from first principles, shows why the answer is almost exactly 1, then shows how batching slides decode up the bandwidth slope toward the ridge. By the end you should be able to derive the result on a whiteboard and explain why a server running a single request wastes more than 99 percent of a modern GPU's compute.
The reason this question shows up in senior and staff interviews is that it separates people who memorise serving tricks from people who understand why those tricks exist. Anyone can say batching helps throughput. The candidates who stand out can show, in two lines of arithmetic, exactly how much it helps and where it stops helping, and can then reason about what changes when you add the KV cache, switch to FP8, or move from H100 to B200. The derivation below is the foundation those conversations are built on.
Counting bytes: the weight read per decode step
A decode step generates exactly one new token. To compute that token's logits, the forward pass multiplies the hidden state through every weight matrix in the model. Each weight must be read from HBM at least once during the step.
For a model with N parameters stored in FP16, each parameter occupies 2 bytes. The total weight read is therefore about 2N bytes per step. This is the denominator of arithmetic intensity, and at batch 1 it is the overwhelming majority of all bytes moved, since the lone token's activations are negligible next to the full weight set.
The critical point: this byte count does not depend on the batch size. Whether you decode for one request or for a hundred, the GPU reads each weight once per step. That single fact is the root of everything that follows.
It is worth being precise about why the read is unavoidable. The weights live in HBM, the large but comparatively slow off-chip memory. The compute happens in the tensor cores, fed through a small on-chip SRAM. There is no room to keep 70 billion parameters resident on chip, so every weight tile must be streamed in from HBM, used for its multiply-accumulate, and discarded. At batch 1 each tile is used exactly once before it is evicted, which is the worst possible reuse pattern. The byte count of 2N is therefore not an upper bound or an estimate; it is the floor that the memory hierarchy forces on you for a single token.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM continuous batching exploits exactly this AI equals B relationship to push decode toward the compute-bound region on H100 and B200 clusters.
- NVIDIA roofline analyses for H100 quote critical intensity near 300, the basis for choosing serving batch sizes for Llama 4 and DeepSeek V4.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does including the KV cache read change the arithmetic intensity at long context?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quoting decode cost in FLOPs and concluding the GPU is compute-limited. Decode at batch 1 uses a tiny fraction of peak FLOPs; the real bottleneck is weight bytes streamed from HBM.
60 second bullets to scan on the way to the call.
The definition of arithmetic intensity as FLOPs per byte moved from memory
Why one decode step reads about 2N bytes of weights in FP16
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.