Spot the errors in this description of continuous batching
Click any words you think contain an error. Click again to unmark.
Continuous batching schedules at the iteration level, evicting and admitting requests every decode step to keep GPU slots full; the win is throughput, and it is the default in online serving.
Picture a carpool van that fixes its riders at the start and refuses to move until every rider reaches their stop. The earliest arrivals still ride along, their empty seats wasted, until the last person is done. That is static batching. Continuous batching is a van that drops off each rider the instant they arrive and immediately picks up someone waiting at the curb. Every seat stays occupied almost all the time. Because short trips finish fast and free up seats, the van moves far more people per hour. The point is not that any single rider arrives sooner. The point is that the van never carries empty seats waiting for one slow passenger.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Continuous batching is the scheduling discipline that lets a single GPU serve many concurrent LLM requests economically. It is also a favourite interview trap, because the naive mental model, that you gather a batch, run it, and start over, describes static batching and is exactly wrong for the continuous case.
The statement under inspection packs three errors into four sentences. It calls continuous batching a static, admission-time-fixed strategy. It claims the goal is reducing time to first token. It confines the technique to offline batch APIs. Each claim inverts the truth. Continuous batching is iteration-level, its win is throughput, and it is the default in online interactive serving.
This deep dive walks through why static batching wastes GPU, how iteration-level scheduling reclaims that waste, why the gain shows up as throughput rather than latency, and how the technique couples to the paged KV cache. By the end you should be able to explain the wasted-slot and head-of-line problems, and say precisely why continuous batching does not pad anything to the longest sequence.
Why static batching wastes the GPU
Static batching fixes the set of requests at admission. The batch runs forward, step after step, until the LAST sequence emits its stop token. Only then does the server form the next batch. The kernel shape is fixed for the batch's whole lifetime, which is convenient to implement but pathological under real traffic.
The problem is that completion lengths are wildly uneven. In a chat workload one request wants 15 tokens and its neighbour wants 900. Under static batching the 15-token request finishes early, but its slot cannot be released until the 900-token sequence is done. The slot sits occupied and idle, burning GPU cycles on nothing. As the batch ages, more and more sequences finish, so average occupancy decays toward a single straggler holding the whole batch hostage.
This is the wasted-slot problem, and it compounds with a second one: head-of-line blocking. A full static batch forces every newly arrived request to wait in the queue until the entire batch drains. A short request that arrives a millisecond after admission can wait seconds for one tail-long sibling. Throughput and latency both suffer.
It is tempting to believe these two strategies are roughly equally efficient and that you simply pick the simpler one. That belief is the misconception this question targets. On any mixed-length workload static batching leaves a large fraction of the GPU's slot-time idle, while continuous batching keeps it near full. The gap is not a few percent; it is routinely a multiple.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM (UC Berkeley) pairs continuous batching with paged attention and is the default high-throughput serving stack across the open ecosystem in 2026.
- Hugging Face Text Generation Inference (TGI) ships continuous batching as its core scheduler for serving Llama 4 and Mistral Large 3 endpoints.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does continuous batching avoid padding requests to the longest sequence?
Scheduling is per iteration, so each sequence simply stops being stepped once it emits its stop token. There is no shared fixed length. The freed KV slot is handed to a waiting request rather than padded out. Contrast this with static batching where the kernel shape is fixed at admission.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming static batching is as efficient as continuous batching, or that continuous batching pads to the longest sequence. Static batching wastes slots on finished requests; continuous batching reclaims them per step.
60 second bullets to scan on the way to the call.
Why static batching wastes GPU slots when completion lengths differ
The defining iteration-level admit and evict loop of continuous batching
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.