What does the vLLM-style continuous-batching scheduler actually do at each step?

**Continuous batching** is the scheduler at the heart of a modern serving engine like vLLM. Its job is to keep the GPU busy by deciding, on every decode step, which requests run together as one batch. Static batching, the old approach, groups a fixed set of requests, pads them to the longest one, and runs the whole group until every request finishes. The problem: a short request that finishes in 20 tokens is stuck waiting for a 2000-token neighbor. The GPU runs the long one nearly alone. That wasted capacity is head of line blocking. Continuous batching fixes this by scheduling at the iteration level. At the start of each step the scheduler reaps any request that just finished, frees its KV cache memory, and admits new waiting requests into the freed space. The binding constraint is KV memory, not compute. The scheduler counts how many free KV blocks exist and admits only as many requests as fit. If memory runs out mid-flight, it preempts a running request, either spilling its cache to CPU or dropping it to rebuild later. Then it runs exactly one batched decode step across every active request and loops. This per-step churn is why vLLM gets several times the throughput of naive serving.

**The continuous-batching scheduler** runs a tight loop once per decode iteration, mixing requests at the token level rather than the request level. It has five responsibilities per step. First, reap finished requests on EOS, a stop sequence, or hitting max tokens, and free their KV blocks. Second, recompute the KV-block budget, namely physical free blocks minus reservations. Third, admit waiting requests up to that budget, running prefill (or chunked prefill for long prompts) for each new admission. Fourth, if memory is over-committed, preempt lower-priority running requests. Fifth, assemble the batch tensor and run one batched decode step. Preemption has two flavors. Swap moves a victim's KV blocks to CPU memory and pages them back over PCIe on resume. Recompute drops the KV and rebuilds it via prefill when the request is readmitted. Recompute wins for short prompts where rebuild is cheap. Swap wins when the prompt is long and CPU bandwidth is available. Throughout, the scheduler tracks block reference counts so shared prefixes use copy on write, plus per-request priority, usually FIFO with starvation guards. The headline is that the scheduler, not paged attention alone, is the throughput lever. A weak admission policy strands KV memory and throws away most of the win.