Explain how continuous batching keeps GPUs busy that static batching leaves idle
An inference server switches from static (request level) batching to continuous (iteration level) batching. Explain the mechanism and why it raises throughput without inflating tail latency.
Continuous batching admits and evicts sequences every decode step, so the GPU stays near-full instead of stalling behind the longest generation — lifting throughput without dragging out short requests.
Imagine a bus that waits at the stop until every passenger reaches their destination before letting anyone new board. A passenger going across town keeps everyone else waiting — including the ones who arrived ages ago. Now imagine the bus lets people off and new people on at every stop. Nobody waits for the long-distance rider to finish, and the bus is always full of passengers actually traveling. That's continuous batching: the GPU swaps finished sequences out and waiting ones in every single step, so it never coasts half-empty.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM serving has a structural awkwardness that no other batched workload shares: you don't know how long each request will run until it's done. A request generates tokens one at a time and stops when the model emits an end of sequence token, so two requests in the same batch can diverge from 20 tokens to 2,000. Batching assumes the members of a batch are roughly interchangeable units of work. Autoregressive generation violates that assumption hard.
Static batching ignores the problem and pays for it in idle GPU cycles. Continuous batching confronts it by changing when the scheduler is allowed to act — from once per batch to once per token step. That one change is the difference between a GPU that coasts half-empty and one that runs near-full while still serving short requests fast.
This deep dive builds the mechanism from the decode loop up: why generation-length variance wrecks static batching, what iteration-level scheduling actually does each step, why it lifts throughput and tail latency together instead of trading one for the other, and the real costs — chunked prefill, KV-cache fragmentation, and fairness — that a senior engineer has to manage to make it work in production.
Why generation-length variance breaks static batching
Static batching, also called request-level batching, treats a batch as one indivisible job. The server gathers N requests, runs the forward loop until every sequence has finished, then forms the next batch. The scheduler acts exactly once per batch.
The trouble is that a batch finishes only when its longest-running member finishes. Suppose 15 of 16 requests generate 20 tokens and one generates 1,000. After step 20, fifteen slots are done — but the batch keeps running for another 980 steps to finish the straggler. During those 980 steps the GPU processes a batch of size one, dressed up as a batch of sixteen.
Two costs stack here. Declining occupancy: the effective batch shrinks token by token as sequences complete, so per-step GPU utilization falls through the batch's life. Head of line blocking: the next batch — including requests that have been waiting in the queue this whole time — can't start until the straggler clears.
And you can't enlarge your way out. A bigger static batch just widens the length distribution it contains, so the gap between when most requests finish and when the batch finishes grows. The structure is wrong, not the size.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | Static batching | Continuous batching |
|---|---|---|
| Scheduling unit | Once per batch | Every decode iteration |
| Batch refill | Only when whole batch finishes | Finished slots refilled mid-flight |
| GPU occupancy | Decays toward the longest request | Stays near-full |
| Short-request latency | Blocked behind long ones | Admitted and completed promptly |
| Main cost | Wasted capacity | Per-step scheduling + KV-cache complexity |
Real products, models, and research that use this idea.
- vLLM ships continuous batching as its default scheduler, paired with PagedAttention for KV-cache management.
- Hugging Face TGI implements continuous (iteration-level) batching for production LLM serving.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does prefill of a newly admitted request interfere with decode of running sequences, and how is that mitigated?
Prefill is a heavy compute burst; chunked prefill splits it across iterations to bound the latency spike on in-flight decode.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking continuous batching just uses a bigger batch, when the real change is scheduling admit/evict decisions every decode iteration instead of once per batch.
60 second bullets to scan on the way to the call.
The scheduling-granularity difference between static and continuous batching
How head of line blocking arises when a batch waits for its longest generation
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.