Predict the arithmetic intensity of decode at varying batch sizes
Single token decode on a model with N parameters, FP16 weights (2 bytes / weight). Ignoring KV cache reads (for a short context this is small relative to weight reads), each decoded token requires: - Reading the entire weight matrix: 2N bytes from HBM - Performing one matmul forward pass: ~2N FLOPs per token With batch size B, ONE weight read serves B parallel token streams; FLOPs scale linearly with B. Compute the arithmetic intensity (FLOPs / bytes read) for B = 1, 8, 64, and 512. Express each as a plain FLOP/byte number.
Decode arithmetic intensity equals the batch size B, because one shared 2N-byte weight read serves 2N*B FLOPs. At small batch this sits far below the hardware ridge point, so decode is memory-bandwidth-bound.
Picture a chef who must walk to a far pantry and haul out every ingredient before cooking a single dish. The walk is slow and fixed; the cooking is quick. If she makes that long trip for just one plate, she spends almost all her time walking, not cooking. So the kitchen mostly sits idle. Now imagine she cooks eight plates from the same single trip to the pantry. The walk costs the same, but the cooking grows eightfold, and she finally spends real time at the stove. The expensive trip is now shared across many plates. The lesson: when fetching the ingredients dominates and the cooking is tiny, the trick is to serve many plates from one trip, so the slow fetch is paid once and the stove stays busy.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Arithmetic intensity is the most useful single number in inference performance reasoning. It is the ratio of useful compute to memory traffic, measured in FLOPs per byte. Pair it with the roofline model and you can predict, on a napkin, whether a workload is starved by the chip's math units or by its memory bandwidth.
The question asks you to compute this ratio for single-token decode at four batch sizes. The setup is deliberately simplified: each step reads the full weight matrix once, costing 2N bytes for an N-parameter FP16 model, and performs a forward matmul costing about 2N FLOPs. The crucial structural fact is that one weight read serves the entire batch, while the FLOP count scales with how many token streams you push through that read.
The arithmetic is short, but the conclusion is the whole game of inference economics. Decode at low batch has an intensity near one. The hardware can do hundreds of FLOPs per byte. So the chip spends almost all its time waiting on memory, and the only way out is to batch. This deep dive derives the ratio, places it on the roofline, and explains why the KV cache ultimately limits how far batching can take you.
Defining arithmetic intensity and the ridge point
Arithmetic intensity is the ratio of operations performed to bytes moved between the compute units and memory:
The roofline model pairs this with two hardware numbers: peak compute in FLOPs per second and peak memory bandwidth in bytes per second. Their ratio is the ridge point, the intensity at which a workload transitions from memory-bound to compute-bound. The model captures a simple physical truth. A processor can only run as fast as the slower of two pipelines: the one feeding it data and the one doing math.
Below the ridge point, you cannot feed the math units fast enough; performance is capped by bandwidth. The achievable throughput is then intensity times bandwidth, a sloped line. Above the ridge, the memory system can keep up and performance is capped by raw compute, a flat ceiling. The two regimes meet at the ridge, which is why the diagram looks like a roof.
On an H100 with roughly 990 BF16 teraFLOPs and about 3.35 terabytes per second of HBM3, the ridge point lands near 295 to 330 FLOPs per byte. That single threshold is the bar every workload is measured against. The exact figure shifts with precision and with the specific board. FP8 roughly doubles peak compute while bandwidth holds, which pushes the ridge point even higher. The headline takeaway does not change: the ridge sits in the hundreds of FLOPs per byte, and any workload far below it is memory-bound by definition.
A crucial subtlety is that arithmetic intensity is a property of the workload, not the hardware. The same kernel can be bandwidth-bound on one chip and compute-bound on another, because the ridge moves. So the right way to reason is to compute the workload intensity first, then ask which side of the ridge it falls on for the chip you actually have.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM exists primarily to raise decode batch size via paged attention and continuous batching, pushing arithmetic intensity up toward the H100 ridge point.
- NVIDIA's H100 datasheet pairs about 990 BF16 teraFLOPs with 3.35 TB/s HBM3, giving a ridge point near 295 to 330 FLOPs per byte that decode must clear.
What an interviewer would ask next. Try answering before peeking at the approach.
QAt what batch size does decode cross from memory-bound to compute-bound on an H100?
Set intensity equal to the ridge point. Since intensity equals B in this model, the crossover is near B equal to the ridge value, roughly 295 to 330. Beyond that, adding batch no longer buys linear throughput because compute saturates.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Concluding decode is compute-bound because it does billions of FLOPs. The FLOP count is large but the byte count is larger relative to compute throughput, so the ratio is tiny and memory wins.
60 second bullets to scan on the way to the call.
The definition of arithmetic intensity as FLOPs divided by bytes read
Why the weight read is shared across the batch but the FLOPs are not
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.