- 1Reap any requests that finished generation in the previous step (EOS, stop sequence, or max_tokens hit) and free their KV blocks
- 2Run one batched decode step over the resulting set of active requests, advancing each by exactly one token
- 3Admit waiting requests up to the budget, performing prefill (or chunked-prefill) for newly admitted ones
- 4Compute the available KV-block budget given current GPU memory and any in-flight reservations
- 5If memory pressure remains, preempt lowest-priority running requests by swapping their KV to CPU or marking them for recompute
Each scheduler iteration runs: reap finished requests and free their KV, compute the block budget, admit waiting requests, preempt under pressure, then run one batched decode step.
Picture a small ferry that crosses a river over and over. Before each crossing, the captain first lets off everyone who reached their stop, freeing up seats. Then he counts how many seats are now open. He waves new passengers aboard up to that count. If he somehow over-promised and the boat is too heavy, he asks the lowest-priority passengers to step off and wait for a later trip. Only then does he actually cross the river once, moving everyone forward by one stop. Next trip, he repeats the whole routine. The crossing itself is always the last thing, because the captain wants the fullest, safest boat before spending the fuel.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: token-level scheduling motivation + the five-step order with the reason for each position + preemption as fallback + why decode is last + paged attention as the enabler.
Real products, models, and research that use this idea.
- vLLM's scheduler implements exactly this iteration-level loop, freeing KV blocks on completion before recomputing the admission budget each step.
- SGLang runs continuous batching with RadixAttention prefix sharing, admitting requests against a live block budget every decode step.
- TensorRT-LLM (NVIDIA) ships in-flight batching, its name for the same per-iteration admit-and-decode loop on H100 and B200 hardware.
- Hugging Face TGI uses continuous batching with token-level scheduling so new requests join the running batch without waiting for it to drain.
- Anthropic and OpenAI serving stacks rely on continuous batching to keep decode-phase GPU utilization high across thousands of concurrent Claude Opus 4.7 and GPT-5.5 sessions.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy must reaping finished requests happen before computing the KV-block budget?
QWhy is preemption placed after admission rather than before it?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Putting the decode step first or admitting new requests before freeing finished ones. You must reclaim KV blocks before you can know the true budget for admission.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.