Describe the responsibilities of a vLLM-style continuous-batching scheduler at each decode iteration. Be specific about admission, eviction, preemption, and what happens when KV memory is over-committed.
Free · 2 AI evals / day
**Continuous batching** is the scheduler at the heart of a modern serving engine like vLLM. Its job is to keep the GPU busy by deciding, on every decode step, which requests run together as one batch.
Static batching, the old approach, groups a fixed set of requests, pads them to the longest one, and runs the whole group until every request finishes. The problem: a short request that finishes in 20 tokens is stuck waiting for a 2000-token neighbor. The GPU runs the long one nearly alone. That wasted capacity is head of line blocking.
Continuous batching fixes this by scheduling at the iteration level. At the start of each step the scheduler reaps any request that just finished, frees its KV cache memory, and admits new waiting requests into the freed space.
The binding constraint is KV memory, not compute. The scheduler counts how many free KV blocks exist and admits only as many requests as fit. If memory runs out mid-flight, it preempts a running request, either spilling its cache to CPU or dropping it to rebuild later.
Then it runs exactly one batched decode step across every active request and loops. This per-step churn is why vLLM gets several times the throughput of naive serving.
**The continuous-batching scheduler** runs a tight loop once per decode iteration, mixing requests at the token level rather than the request level.
It has five responsibilities per step. First, reap finished requests on EOS, a stop sequence, or hitting max tokens, and free their KV blocks. Second, recompute the KV-block budget, namely physical free blocks minus reservations. Third, admit waiting requests up to that budget, running prefill (or chunked prefill for long prompts) for each new admission. Fourth, if memory is over-committed, preempt lower-priority running requests. Fifth, assemble the batch tensor and run one batched decode step.
Preemption has two flavors. Swap moves a victim's KV blocks to CPU memory and pages them back over PCIe on resume. Recompute drops the KV and rebuilds it via prefill when the request is readmitted. Recompute wins for short prompts where rebuild is cheap. Swap wins when the prompt is long and CPU bandwidth is available.
Throughout, the scheduler tracks block reference counts so shared prefixes use copy on write, plus per-request priority, usually FIFO with starvation guards.
The headline is that the scheduler, not paged attention alone, is the throughput lever. A weak admission policy strands KV memory and throws away most of the win.
Quick reference
TL;DR
Each iteration the scheduler reaps finished requests, recomputes the KV-block budget, admits waiting work up to it, preempts under pressure via swap or recompute, then runs one batched decode step.
Picture a busy restaurant kitchen with one shared stove. Every few seconds the head chef checks it. Finished dishes come off and free up burners. New orders waiting at the pass get started, but only as many as the freed burners can hold. If the kitchen is jammed and one long order is hogging space, the chef sets a dish aside, either keeping its half-cooked food warm in the fridge to resume later, or scraping it and re-prepping from scratch when room opens up. Then the chef pushes every active dish one step forward together, in a single sweep, before checking again. The stove is the GPU, the burners are KV memory, and that constant per-step reshuffle is continuous batching.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-request offline inference with no concurrency at all With one request there is nothing to batch across, so scheduler overhead buys nothing over a plain decode loop.
Tight, uniform latency targets where every request must finish in lockstep Preemption introduces tail-latency variance; a fixed static batch gives more predictable per-request timing.
Workloads of nearly identical prompt and output lengths Without length variance there is little head of line blocking, so static batching captures most of the throughput already.
Severely memory-constrained deployment that cannot afford swap space or recompute Preemption needs either CPU memory for swap or spare compute for rebuild; with neither, admit conservatively instead.
Quick scan · 2–4 min
Everything important, quickly.
4 min: the five per-step responsibilities, reap-before-admit and head-of-line blocking, KV-block budget as the admission gate, swap versus recompute preemption, then chunked prefill and prefix-sharing refcounts.
Real-world examples
Real products, models, and research that use this idea.
1vLLM (UC Berkeley) pairs continuous batching with paged attention and is the reference scheduler most open-source serving builds on in 2026.
2SGLang adds prefix-aware scheduling so requests sharing a long system prompt are admitted together and reuse cached blocks.
3TensorRT-LLM (NVIDIA) ships in-flight batching, its name for the same iteration-level scheduler, with chunked prefill on H100 and B200.
4Hugging Face TGI implements continuous batching with a token-budget scheduler for serving Llama 4 and Mistral Large 3.
5Anthropic and OpenAI production stacks combine iteration-level scheduling with prefix caching so common system prompts skip prefill across requests.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does decode push a batch into over-commit even when admission was within budget?
Sign in or go Pro to see the approach for this follow-up.
QHow does the scheduler decide between swap and recompute for a preemption victim?
Sign in or go Pro to see the approach for this follow-up.
QWhat does chunked prefill solve that plain prefill-then-decode does not?
Sign in or go Pro to see the approach for this follow-up.
QWhy are block reference counts central to safe prefix sharing?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Describing it as static batching that pads to the longest sequence. The whole point is iteration-level scheduling, where finished requests leave and new ones join every step, with no head of line blocking.