What is the arithmetic intensity of an LLM decode step and why is it close to 1?

Attempt it

Derive the arithmetic intensity (FLOPs per byte read from HBM) of one decode step at batch 1 and at batch B, ignoring the KV cache contribution. Use a model with N parameters in FP16. Explain what the result implies about decode's place on the roofline.

Free · 2 AI evals / day

Quick reference

TL;DR

At batch 1 a decode step reads 2N weight bytes and does 2N FLOPs, so arithmetic intensity is 1 FLOP per byte. Batching to B raises it to B, sliding decode up the roofline toward compute-bound.

Memory aid

Easy to grasp

Imagine a chef who must walk to a giant pantry, carry back every ingredient, then cook one tiny dish. The walk is the slow part, not the cooking. Serving one customer, almost all the time is spent fetching, so the stove sits nearly idle. Now imagine forty customers ordering at once. The chef makes one trip to the pantry and cooks forty dishes from the same haul. The walk cost is shared, so the stove finally runs hot. LLM decode is the same. Reading the model weights from memory is the long walk; the math is the quick cooking. One request wastes the hardware; batching many requests pays for the trip once and keeps the chips busy.

Key concepts

Arithmetic intensity Roofline model Batching Memory-bound decode

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: define arithmetic intensity, derive 2N bytes and 2N FLOPs at batch 1 giving AI=1, show batching cancels to AI=B, compute the H100 critical intensity near 330, then place decode on the roofline and connect to throughput.

Real-world examples

Real products, models, and research that use this idea.

vLLM continuous batching exploits exactly this AI equals B relationship to push decode toward the compute-bound region on H100 and B200 clusters.
NVIDIA roofline analyses for H100 quote critical intensity near 300, the basis for choosing serving batch sizes for Llama 4 and DeepSeek V4.
TensorRT-LLM tunes max batch and in-flight batching specifically to climb the bandwidth slope without exceeding KV cache HBM limits.
Together AI and Modal publish throughput versus batch curves that show the linear intensity growth flattening as decode crosses into compute-bound.
Anthropic and OpenAI batch many concurrent Claude Opus 4.7 and GPT-5.5 requests per replica to amortise the per-step weight read across users.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does including the KV cache read change the arithmetic intensity at long context?

QCompute the critical arithmetic intensity for an H100 and explain what it means.

QIf intensity equals B, why not just set the batch size enormous?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Quoting decode cost in FLOPs and concluding the GPU is compute-limited. Decode at batch 1 uses a tiny fraction of peak FLOPs; the real bottleneck is weight bytes streamed from HBM.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.