Why is batching the single biggest throughput lever for LLM decode?

Attempt it

Explain mechanically why batching N concurrent requests during decode raises GPU throughput far more than batching does for, say, a CNN classifier. Where does the linear scaling start to break down?

Free · 2 AI evals / day

Quick reference

TL;DR

Decode is memory-bound, so one weight read per step is wasted on a single token. Batching amortizes that read across many tokens, raising throughput near-linearly until the compute roof.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a chef who must walk to a huge pantry and haul out every ingredient just to cook one tiny dish. The walk is the slow part, not the cooking. If only one order is on the ticket, that long walk feeds a single plate. But if twenty orders are waiting, the chef hauls the same ingredients once and plates all twenty in roughly the same trip. The walk to the pantry is reading the model weights from memory. The cooking is the actual math. Because the walk dominates, serving twenty requests together is almost as cheap as serving one. That is why batching is the biggest single throughput win for language model serving. It pays the expensive memory trip once and spreads it across many answers.

Key concepts

Batching Arithmetic intensity Roofline model Continuous batching

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: decode is memory-bound at batch 1 + weight-read amortization across the batch + arithmetic intensity on the roofline + critical batch B-star + continuous batching + throughput-latency tension.

Real-world examples

Real products, models, and research that use this idea.

vLLM continuous batching keeps the decode batch full by scheduling at the token level, delivering several times the throughput of static batching on Llama 4 serving.
NVIDIA TensorRT-LLM exposes in-flight batching plus a tunable max batch size so operators can sit near the critical batch on H200 and B200 hardware.
Anthropic and OpenAI run high-concurrency batched decode behind their APIs, which is why per-token prices fall as aggregate traffic rises.
SGLang batches DeepSeek V4 and Qwen 3 requests with prefix sharing, raising effective batch size for workloads with common system prompts.
The roofline model from Berkeley is the standard tool engineers use to locate the ridge point that sets a model's critical batch on a given GPU.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you estimate the critical batch B-star for a given model and GPU?

Sign in or go Pro to see the approach for this follow-up.

QWhy does continuous batching beat static batching even when both target the same max batch?

Sign in or go Pro to see the approach for this follow-up.

QHow does the KV cache interact with the batch-size throughput story?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming batching helps because the GPU has spare compute. The real reason is amortizing the weight read in a memory-bound regime; spare compute is the symptom, not the cause.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.