Why did continuous batching replace static batching for LLM serving?

Decode runs one token at a time, and a single request barely uses a GPU. Batching many requests together amortizes the cost of reading the model weights from memory across all of them, so batching is the biggest throughput lever in serving. **Static batching** forms a batch, runs it, and waits for every sequence in it to finish before starting the next batch. The problem is that completion lengths vary wildly. A chat reply might be 20 tokens while a summary is 800. The short ones finish early, but their slot in the batch sits empty until the single longest sequence terminates. That idle time is wasted GPU. On real traffic the average slot is busy only a fraction of the time, so most of an expensive accelerator does nothing. **Continuous batching** schedules at every decode step instead of once per request. At the start of each step it evicts any finished sequence and admits a waiting request into the freed slot. The batch contents change constantly, but the batch stays full. Because every slot does useful work almost every step, end-to-end throughput climbs roughly 10 to 20 times on production traces. The price is a more complex scheduler that tracks admission, eviction, and memory budget.

**Static batching** schedules at request granularity. It forms a batch, decodes until every sequence in it finishes, then forms the next batch. Completion lengths vary wildly. A short reply may be 20 tokens while a long one is 800. The short sequences finish early, yet their slot stays idle until the longest sequence terminates. That is head-of-line blocking, and it wrecks utilization. On mixed traffic the average slot is busy only a small fraction of the time. Continuous batching schedules at iteration granularity instead. Before each decode step it evicts finished sequences and admits waiting requests into the freed slots. The batch composition changes every step, but the GPU runs a full batch nearly every step. The win is that scheduling granularity drops from a whole completion lifetime to a single token. No slot waits on the longest sequence. Reported throughput rises roughly 10 to 20 times on production traces, which is why vLLM, Orca, and TGI all adopted it. The cost is scheduler complexity. The system must track per-step admission, eviction, and a KV-cache memory budget, and preempt sequences when memory runs short.