What is the primary reason batching is so much more effective for LLM decode than for CNN inference?

Attempt it

Quick reference

TL;DR

LLM decode is memory-bandwidth-bound at batch 1, so batching amortizes the weight read across many tokens; a CNN is already compute-bound and has little to amortize.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a librarian who must walk to a huge back room to fetch one heavy reference book every time someone asks a question. If one person is waiting, the librarian makes the long walk for a single answer. That walk is the cost of loading the model weights from memory. Now imagine ten people ask questions that all need the same book. The librarian makes one walk and answers all ten at once. The walk cost is shared across everyone. That is batching for LLM decode. A CNN is different, more like a chef who already chops a whole crate of vegetables in one motion. The chef stays busy the whole time, so adding more orders barely helps because the work was already packed tight.

Key concepts

Batching Roofline model Arithmetic intensity Memory-bandwidth-bound

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: roofline regime at batch 1 + why decode is bandwidth-bound + how batching raises arithmetic intensity + saturation at the compute roof + KV cache and continuous batching caveats.

Real-world examples

Real products, models, and research that use this idea.

vLLM uses continuous batching plus paged attention to keep batch sizes high during decode, the main reason it beats naive serving by 5 to 10 times on Llama 4.
NVIDIA TensorRT-LLM tunes the batch sweet spot on H100 and B200 where decode crosses from bandwidth-bound to compute-bound.
OpenAI and Anthropic batch many concurrent chat requests through one GPU replica so the weight read for GPT-5.5 or Claude Opus 4.7 decode is amortized.
DeepSeek V4 pairs Multi-head Latent Attention with large decode batches so the shrunken KV cache leaves bandwidth headroom for bigger batches.
SGLang schedules requests at token granularity, sustaining large effective batches across requests that share long system prompts.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QAt what batch size does LLM decode stop benefiting from more batching?

Sign in or go Pro to see the approach for this follow-up.

QHow does the KV cache change the batching analysis at long context?

Sign in or go Pro to see the approach for this follow-up.

QWhy is prefill compute-bound while decode is bandwidth-bound on the same model?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Attributing the batching win to LLMs having more parameters. The driver is the bottleneck regime, not model scale; a memory-bound workload has a weight read to amortize, a compute-bound one does not.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.