As batch size grows, where does throughput stop increasing and per request latency start exploding?
Describe the throughput and TPOT (time per output token) curves as batch size B grows for LLM decode. Identify the breakpoint and explain what happens to throughput and to per request latency before and after it.
Decode throughput rises almost linearly with batch until the critical batch B*, then plateaus; past B* per-token latency rises linearly while throughput buys you nothing.
Imagine a bus that drives the same route in the same time no matter how many riders are aboard. With one rider the trip is wasteful, since all the fuel is spread over a single person. Add more riders and each ride gets cheaper, because that fixed driving cost is now shared. People delivered per hour keeps climbing. But the bus has only so many seats. Once every seat is full, cramming in more cannot make the bus go faster, so the number of people delivered per hour stops rising. Worse, the extra people now have to wait for room before they ever get moving, so each person's door to door time grows longer and longer. The smart move is to fill the bus right up to its seats, and no further.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Batching is the single biggest throughput lever in LLM serving, and the saturation curve is the reason it has a ceiling. Senior interviews on inference optimization almost always probe whether you understand that ceiling quantitatively, not just that batching helps. The naive belief that more batch is always better is exactly the trap the question is built to catch.
The whole story lives on the roofline. Decode starts memory-bound, so batching amortizes a fixed memory cost and throughput climbs. At a precise batch the workload becomes compute-bound, throughput flatlines, and the cost of more batch shifts entirely onto per-request latency. The two curves, throughput and time per output token, trade roles at that hinge. Before it they have opposite shapes; after it they swap.
The reason both curves bend at the same point is that they are two views of one transition. Throughput measures the system; time per output token measures the individual request. The same resource switch, from bandwidth-bound to compute-bound, drives both. Once you internalize that, the entire serving picture becomes a single graph with one critical x-coordinate.
This deep dive builds the picture from first principles. It explains why decode is memory-bound, how batching changes arithmetic intensity, where the critical batch comes from on the roofline, what happens to each curve on either side of the knee, and how production schedulers actually choose an operating point that is rarely the peak-throughput batch.
Why decode is memory-bound at small batch
Autoregressive decode produces one new token per request per forward pass. To compute that token the GPU must stream every weight matrix of the model out of high-bandwidth memory. For a dense model that is the entire parameter set, read once per step. A 70B model in bf16 means about 140 GB of weights crossing the memory bus on every single decode step.
The arithmetic attached to a single token is tiny relative to the bytes moved. A matrix times a single vector is a GEMV: it touches every weight but does only one multiply-add per weight. The ratio of FLOPs to bytes, the arithmetic intensity, is therefore very low, on the order of one or two operations per byte. Compare that to the hundreds of operations per byte the hardware can sustain at peak and the imbalance is stark.
Low intensity means the tensor cores sit idle waiting on memory. The step time is set by how fast you can read weights, not by how fast you can multiply. This is the defining condition of the memory-bound regime. It is why a single-request decode wastes most of an expensive accelerator: the chip's headline FLOPs number is irrelevant when the bottleneck is the trip to HBM. This is also the central misconception the concept warns against, optimizing for FLOPs when the workload is bound by memory bandwidth.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM continuous batching admits requests at the token level and targets the throughput knee while honoring a TPOT service level objective.
- NVIDIA H100 roofline analysis places the decode ridge point at several hundred tokens of work, the standard reference for sizing B* in 2026.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does time per output token stay roughly flat below B* but not perfectly flat?
Separate the two memory terms. Weight reads are shared across the batch and amortize cleanly. KV-cache reads are per request and grow with both batch and sequence length, so they add a small slope to TPOT even in the memory-bound region.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming throughput keeps rising with batch forever, or that latency is flat. Past the critical batch, throughput is pinned and every extra request only inflates per-token latency.
60 second bullets to scan on the way to the call.
Why decode is memory-bound at small batch and what the fixed cost is
How batching amortizes the HBM weight read across requests
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.