Predict the arithmetic intensity of decode at varying batch sizes

**Arithmetic intensity** is a single ratio: how many math operations you do per byte you move from memory. It tells you whether a workload is limited by the chip's compute or by its memory bandwidth. In LLM decode at batch size one, each new token requires reading every weight in the model once. That read is 2N bytes for an N-parameter FP16 model. The matmul that consumes those weights costs about 2N FLOPs. So the ratio is roughly two FLOPs per byte read, which cancels down to about one. That is a very small number. Why small matters: a modern GPU can do hundreds of FLOPs in the time it takes to read one byte from HBM. The break-even point, called the ridge point, sits around 330 FLOPs per byte on an H100. Decode at intensity one is nowhere near that line. The consequence is that decode is memory-bandwidth-bound. The compute units sit mostly idle, waiting on weight reads. The fix is batching. Read the weights once, then run many token streams through them. The intensity climbs to the batch size, and only past a few hundred concurrent streams does decode finally become compute-bound.

**Arithmetic intensity** is the ratio of compute to memory traffic. The definition is: $$\text{AI} = \frac{\text{FLOPs}}{\text{bytes read}}$$ For single-token decode the dominant cost is reading the weights. An N-parameter FP16 model reads 2N bytes per step. The forward matmul costs about 2N FLOPs. At batch one that gives an intensity near one FLOP per byte. With batch size B, the same 2N-byte weight read serves all B streams. FLOPs become 2N times B. The 2N terms cancel, leaving intensity equal to B. So the four answers are 1, 8, 64, and 512. The ridge point matters next. On an H100 the critical intensity is about 330. Decode at small batch sits far below that line. It is memory-bandwidth-bound, so the tensor cores idle while HBM streams weights. Batching is the lever. Intensity rises linearly with B until it crosses the ridge point near a few hundred streams. After that, decode turns compute-bound and throughput plateaus. This is why serving stacks chase the largest batch the KV cache budget allows.