As batch size grows, where does throughput stop increasing and per-request latency start exploding?

Attempt it

Describe the throughput and TPOT (time-per-output-token) curves as batch size B grows for LLM decode. Identify the breakpoint and explain what happens to throughput and to per-request latency before and after it.

Free · 2 AI evals / day

Quick reference

TL;DR

Decode throughput rises almost linearly with batch until the critical batch B*, then plateaus; past B* per-token latency rises linearly while throughput buys you nothing.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a bus that drives the same route in the same time no matter how many riders are aboard. With one rider the trip is wasteful, since all the fuel is spread over a single person. Add more riders and each ride gets cheaper, because that fixed driving cost is now shared. People delivered per hour keeps climbing. But the bus has only so many seats. Once every seat is full, cramming in more cannot make the bus go faster, so the number of people delivered per hour stops rising. Worse, the extra people now have to wait for room before they ever get moving, so each person's door to door time grows longer and longer. The smart move is to fill the bus right up to its seats, and no further.

Key concepts

Batching Roofline model Arithmetic intensity Time per output token

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: memory-bound decode + batching amortizes the weight read + roofline ridge point and B* formula + throughput plateau and TPOT climb past the knee + how continuous batching picks the operating point.

Real-world examples

Real products, models, and research that use this idea.

vLLM continuous batching admits requests at the token level and targets the throughput knee while honoring a TPOT service level objective.
NVIDIA H100 roofline analysis places the decode ridge point at several hundred tokens of work, the standard reference for sizing B* in 2026.
TensorRT-LLM in-flight batching and chunked prefill on H100 and B200 balance decode batch against the latency budget for production serving.
SGLang schedules Llama 4 and Qwen 3 decode batches against measured TPOT curves rather than a single static batch size.
Together AI and Modal expose max-batch and latency knobs that map directly to operating below the saturation batch B*.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does time per output token stay roughly flat below B* but not perfectly flat?

Sign in or go Pro to see the approach for this follow-up.

QHow does the critical batch shift when you move from bf16 to fp8 weights and KV cache?

Sign in or go Pro to see the approach for this follow-up.

QHow does long context change where the knee sits?

Sign in or go Pro to see the approach for this follow-up.

QIf throughput is already at the roof, what levers raise it without touching batch?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming throughput keeps rising with batch forever, or that latency is flat. Past the critical batch, throughput is pinned and every extra request only inflates per-token latency.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.