Spot the errors in this 'batching is universal' claim

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Batching helps because decode is memory-bound: at batch 1 the GPU streams weights from HBM and barely computes, so adding requests amortizes one weight load over many tokens.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a delivery truck driving across town to hand over one envelope, then driving back empty. The drive is the expensive part; the envelope weighs nothing. That round trip is loading the model weights from memory. Doing it for a single token wastes the whole journey. Batching is loading the truck with envelopes for many addresses on one trip. The drive costs the same whether you carry one envelope or fifty, so the cost per delivery plummets. LLM decode is exactly this: reading the weights dominates, the math per token is tiny. Pack more requests into one weight read and throughput soars. But a workload that already fills the truck every trip gains nothing from a bigger truck.

Key concepts

Batching Roofline model Arithmetic intensity Memory-bandwidth-bound

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: roofline regimes + why decode is a memory-bound GEMV at batch 1 + how batching amortizes the weight load into a GEMM + why prefill and CNN inference gain little + the KV cache batch ceiling.

Real-world examples

Real products, models, and research that use this idea.

vLLM continuous batching is the headline decode throughput lever; it amortizes weight reads across requests and drives utilization toward the compute roof.
SGLang batches Llama 4 and Qwen 3 decode steps so a single HBM weight read serves dozens of concurrent sequences.
NVIDIA TensorRT-LLM uses in-flight batching on H100 and B200 precisely because decode GEMV kernels are HBM-bandwidth-bound at batch 1.
DeepSeek V4 serving stacks pair large decode batches with MLA so the KV cache does not eat the memory headroom that batching needs.
A ResNet-50 image classifier on an H100 is compute-bound at batch 1; doubling the batch mostly queues work and barely lifts throughput.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does batching turn a memory-bound decode step into a compute-bound one?

Sign in or go Pro to see the approach for this follow-up.

QWhat caps the maximum useful decode batch size in production?

Sign in or go Pro to see the approach for this follow-up.

QHow would you decide whether batching will help a new workload before measuring?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming batching helps by adding more compute or because the workload is compute-bound. Decode is memory-bound at batch 1; batching wins by amortizing one weight load over many tokens, not by adding FLOPs.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.