Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
**Batching** runs several requests through the GPU together so one expensive read of the model weights is shared across many sequences. The question is when you fix which requests are in the batch.
Static batching fixes the batch at admission. All requests start together and the batch runs until the LAST one finishes. The catch: completions have wildly different lengths. A request that wanted 20 tokens sits in a finished, idle slot while a neighbour grinds out 800 tokens. Those idle slots are pure wasted GPU.
Continuous batching fixes this by scheduling at the iteration level. After every single decode step, the server checks which requests finished, evicts them, and pulls waiting requests into the freed slots. The batch composition is fluid: requests join and leave constantly.
The payoff is throughput. The GPU stays packed with live work instead of holding empty slots behind one long sequence. That is why every modern serving stack, vLLM, TGI, and TensorRT-LLM, makes continuous batching the default rather than an option.
The statement in the question inverts all of this: it calls continuous batching static, credits it with reducing TTFT, and confines it to offline APIs. All three are wrong.
**Continuous batching** is iteration-level scheduling, the opposite of the static description given. After every decode step the server evicts finished requests and admits waiting ones into the freed slots. Batch composition changes constantly.
Static batching, by contrast, fixes the batch at admission. It runs until the longest sequence in the batch finishes. Completion lengths vary by an order of magnitude. So short requests leave their slots idle while one tail-long sequence drags on. That idle GPU time is the waste continuous batching removes.
The primary win is throughput, not TTFT. Iteration-level scheduling keeps every slot doing live work. It does not make any single request start faster. TTFT is governed by prefill scheduling and queue depth, separate concerns. Crucially, continuous batching does NOT pad to the longest sequence. Each request runs exactly its own number of steps.
It is the default in online serving, not offline batch. Online chat is precisely where short and long completions intermix. Static batching would waste capacity there catastrophically. vLLM, TGI, and TensorRT-LLM all ship it as the standard scheduler.
Quick reference
TL;DR
Continuous batching schedules at the iteration level, evicting and admitting requests every decode step to keep GPU slots full; the win is throughput, and it is the default in online serving.
Picture a carpool van that fixes its riders at the start and refuses to move until every rider reaches their stop. The earliest arrivals still ride along, their empty seats wasted, until the last person is done. That is static batching. Continuous batching is a van that drops off each rider the instant they arrive and immediately picks up someone waiting at the curb. Every seat stays occupied almost all the time. Because short trips finish fast and free up seats, the van moves far more people per hour. The point is not that any single rider arrives sooner. The point is that the van never carries empty seats waiting for one slow passenger.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single request served alone with no concurrent traffic to fill a batch With nothing else to admit there are no idle sibling slots to reclaim, so iteration scheduling adds bookkeeping for no throughput gain.
Strict per request tail latency targets under bursty load Continuous batching optimizes aggregate throughput, not the latency of any single request, which can still queue behind admission limits.
Uniform fixed length generation where every request emits the same token count Static batching wastes nothing when all sequences finish together, so the extra scheduler machinery buys little.
Severely memory constrained deployments with no room for variable cache pressure Variable batch composition needs a paged cache and a memory budget; without headroom the system preempts and thrashes.
Quick scan · 2–4 min
Everything important, quickly.
3 min: static versus continuous batching + the iteration-level admit and evict loop + why the win is throughput not TTFT + no padding to longest + paged cache interaction.
Real-world examples
Real products, models, and research that use this idea.
1vLLM (UC Berkeley) pairs continuous batching with paged attention and is the default high-throughput serving stack across the open ecosystem in 2026.
2Hugging Face Text Generation Inference (TGI) ships continuous batching as its core scheduler for serving Llama 4 and Mistral Large 3 endpoints.
3NVIDIA TensorRT-LLM implements in-flight batching, its name for continuous batching, alongside chunked prefill on H100 and B200 hardware.
4SGLang combines continuous batching with RadixAttention prefix sharing to serve Qwen 3 and DeepSeek V4 at high request concurrency.
5The original Orca paper introduced iteration-level scheduling, the idea that became continuous batching in modern production serving systems.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does continuous batching avoid padding requests to the longest sequence?
Sign in or go Pro to see the approach for this follow-up.
QWhy does continuous batching help throughput but not necessarily TTFT?
Sign in or go Pro to see the approach for this follow-up.
QHow do continuous batching and paged KV cache depend on each other?
Sign in or go Pro to see the approach for this follow-up.
QWhat is head-of-line blocking and how does iteration-level scheduling remove it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming static batching is as efficient as continuous batching, or that continuous batching pads to the longest sequence. Static batching wastes slots on finished requests; continuous batching reclaims them per step.