Why does batching help LLM decode disproportionately more than batching helps CNN inference?

Attempt it

Explain why a typical CNN classifier gets marginal throughput wins from batching while LLM decode sees ~10× or more. Tie the answer to where each workload sits at batch 1.

Free · 2 AI evals / day

Quick reference

TL;DR

LLM decode at batch 1 is memory bound, so batching amortizes one weight read across many requests and throughput climbs nearly linearly. CNNs start compute bound, so batching just queues work.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a chef who must walk to a far pantry, carry one heavy sack of flour back, and bake just one loaf with it. The walk dominates the time. If one customer orders a single loaf, the chef wastes the whole trip on that one loaf. But if ten customers each want a loaf, the chef makes one trip and bakes ten loaves from the same sack. The slow walk is paid once and shared across ten orders, so output jumps almost tenfold with barely more time. Now picture a different cook whose counter is already packed and whose hands never stop moving. Giving that cook ten orders does not help, because the hands were already the bottleneck. LLM decode is the first chef; the pantry walk is reading weights from memory.

Key concepts

Batching Roofline model Arithmetic intensity Memory bandwidth bound

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: roofline model + arithmetic intensity + CNN compute bound versus decode memory bound at batch 1 + how batching amortizes weight reads + the critical batch size + continuous batching in production.

Real-world examples

Real products, models, and research that use this idea.

vLLM uses continuous batching to push decode batch size toward the critical point, delivering 5 to 10x throughput over naive request-level serving in 2026.
Serving Llama 3.1 70B on an H100, decode at batch 1 reads roughly 140 GB of bf16 weights per token and is HBM bound, while batch 64 amortizes that read across 64 sequences.
NVIDIA's TensorRT-LLM pairs in-flight batching with paged KV cache so B200 deployments keep decode near the compute roof.
SGLang schedules many requests sharing a system prompt into one batch, amortizing both weight reads and prefix KV across the group.
DeepSeek V4 serving relies on large decode batches to make its mixture-of-experts weight reads economical per token.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat sets the critical batch size B* where decode crosses from memory bound to compute bound?

Sign in or go Pro to see the approach for this follow-up.

QWhy does the KV cache complicate the simple weight-amortization story for batching?

Sign in or go Pro to see the approach for this follow-up.

QHow does continuous batching beat static batching for real serving throughput?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming batching helps every workload equally, or that it helps because the GPU likes bigger matrices. It only helps when arithmetic intensity has headroom before the compute roof.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.