Compare static (request-level) batching with continuous (iteration-level) batching for LLM serving. Why does the switch deliver ~10-20× throughput in practice? Be specific about what happens at the end of a short sequence inside a batch.
Free · 2 AI evals / day
Decode runs one token at a time, and a single request barely uses a GPU. Batching many requests together amortizes the cost of reading the model weights from memory across all of them, so batching is the biggest throughput lever in serving.
**Static batching** forms a batch, runs it, and waits for every sequence in it to finish before starting the next batch. The problem is that completion lengths vary wildly. A chat reply might be 20 tokens while a summary is 800. The short ones finish early, but their slot in the batch sits empty until the single longest sequence terminates.
That idle time is wasted GPU. On real traffic the average slot is busy only a fraction of the time, so most of an expensive accelerator does nothing.
**Continuous batching** schedules at every decode step instead of once per request. At the start of each step it evicts any finished sequence and admits a waiting request into the freed slot. The batch contents change constantly, but the batch stays full.
Because every slot does useful work almost every step, end-to-end throughput climbs roughly 10 to 20 times on production traces. The price is a more complex scheduler that tracks admission, eviction, and memory budget.
**Static batching** schedules at request granularity. It forms a batch, decodes until every sequence in it finishes, then forms the next batch. Completion lengths vary wildly. A short reply may be 20 tokens while a long one is 800. The short sequences finish early, yet their slot stays idle until the longest sequence terminates.
That is head-of-line blocking, and it wrecks utilization. On mixed traffic the average slot is busy only a small fraction of the time.
Continuous batching schedules at iteration granularity instead. Before each decode step it evicts finished sequences and admits waiting requests into the freed slots. The batch composition changes every step, but the GPU runs a full batch nearly every step.
The win is that scheduling granularity drops from a whole completion lifetime to a single token. No slot waits on the longest sequence. Reported throughput rises roughly 10 to 20 times on production traces, which is why vLLM, Orca, and TGI all adopted it.
The cost is scheduler complexity. The system must track per-step admission, eviction, and a KV-cache memory budget, and preempt sequences when memory runs short.
Quick reference
TL;DR
Static batching holds every slot hostage to the longest sequence; continuous batching evicts finished requests and admits new ones every decode step, keeping the GPU full and lifting throughput 10-20x.
Imagine a tour bus that only leaves once everyone aboard reaches their stop. Some riders get off after one block, but their seats stay empty because the bus waits for the last passenger riding to the far end of town. The whole bus runs half-empty for most of the trip. Continuous batching is a smarter bus: the moment a rider gets off, the driver lets a person waiting at the curb hop into that empty seat. The bus is always full, so every seat does useful work the entire trip. In serving terms, each empty seat is a wasted slice of a very expensive GPU, and keeping every seat filled is what makes inference cheap enough to run as a product.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-request or very low-concurrency local inference With one sequence in flight there are no idle peer slots to refill, so iteration-level scheduling adds overhead with no utilization gain.
Offline batch jobs where every sequence has a similar length When completion lengths are uniform there is little head of line blocking, so static batching already keeps slots busy.
Extremely memory-constrained deployment with no paged KV support Without paged allocation the scheduler cannot pack freed lanes, so mid-flight admission fragments memory and the gain evaporates.
Strict per-request latency isolation requirements Sharing a rolling batch couples tail latency across tenants; admission and preemption can perturb a request that needs guaranteed timing.
Quick scan · 2–4 min
Everything important, quickly.
3 min: batching as a memory-bandwidth win + static batching's head-of-line blocking + iteration-level scheduling (admit/evict per step) + paged KV dependency + scheduler costs.
Side-by-side
Dimension
Static (request-level) batching
Continuous (iteration-level) batching
Scheduling unit
Once per batch of requests
Once per decode step
Batch lifetime
Until the longest sequence finishes
Rolling window, sequences enter and leave each step
Idle slots
Short sequences leave empty slots until the batch ends
Freed slots refilled immediately from the queue
Effective batch size
Decays as sequences finish
Stays near the configured maximum
Throughput on mixed traffic
Baseline
Roughly 10 to 20 times higher
Scheduler complexity
Simple, run to completion
Admission, eviction, KV budget, preemption
KV-cache dependency
Contiguous per-request allocation suffices
Needs paged allocation to admit mid-flight
Real-world examples
Real products, models, and research that use this idea.
1vLLM pairs continuous batching with paged attention as the default open-source serving stack in 2026, packing lanes densely across requests.
2Orca first introduced iteration-level scheduling, the academic origin of continuous batching now standard across serving engines.
3Hugging Face Text Generation Inference (TGI) ships continuous batching to serve Llama 4 and Mistral Large 3 endpoints at high throughput.
4NVIDIA TensorRT-LLM implements in-flight batching, its name for continuous batching, with chunked prefill on H100 and B200 hardware.
5SGLang combines continuous batching with prefix sharing so concurrent requests reusing a long system prompt stay densely packed.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does continuous batching depend on paged attention to work well?
Sign in or go Pro to see the approach for this follow-up.
QHow do prefill and decode contend in a continuous-batching scheduler?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens when the KV-memory budget is exhausted mid-batch?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Claiming static batching wastes compute because of padding. The real loss is idle slots: short sequences finish early and their slot stays empty until the longest sequence in the batch terminates.