1Reap any requests that finished generation in the previous step (EOS, stop sequence, or max_tokens hit) and free their KV blocks
2Run one batched decode step over the resulting set of active requests, advancing each by exactly one token
3Admit waiting requests up to the budget, performing prefill (or chunked-prefill) for newly admitted ones
4Compute the available KV-block budget given current GPU memory and any in-flight reservations
5If memory pressure remains, preempt lowest-priority running requests by swapping their KV to CPU or marking them for recompute
**Continuous batching** is how modern LLM servers keep the GPU busy. Instead of waiting for a whole batch of requests to finish, the scheduler rebuilds the active batch every single decode step. Requests can join and leave mid-flight.
Each iteration follows a fixed routine. First, the scheduler reaps any request that finished in the previous step, because it hit an end-of-sequence token, a stop string, or its max token limit. Freeing those requests releases their KV cache blocks back to the pool.
Second, it computes the budget: how many KV blocks are now free given GPU memory and any reservations. Third, it admits waiting requests up to that budget, running prefill for the newcomers so their prompts are cached.
Fourth, if admission over-committed and memory pressure remains, it preempts the lowest-priority running requests, swapping their cache to CPU or marking them for recompute later. Finally, it runs one batched decode step, advancing every active request by exactly one token. Then the loop repeats.
**Continuous batching** schedules at the token level, not the request level. The scheduler runs a tight loop, and each iteration follows a strict order so that resources are freed before they are handed out.
The order is: reap, budget, admit, preempt, decode. First, reap requests that finished in the previous step and free their KV blocks. Second, compute the available KV-block budget from current GPU memory. Third, admit waiting requests up to that budget, doing prefill or chunked prefill for newcomers. Fourth, if pressure remains after admission, preempt the lowest-priority running requests by swapping their cache to CPU or scheduling recompute. Fifth, run one batched decode step that advances every active request by a single token.
The ordering logic is a classic memory-manager pattern. You must free before you can budget. You must budget before you can admit. Preemption is a fallback that only fires when admission over-committed. The decode step is last because the scheduler wants the densest safe batch before it spends the expensive forward pass.
Quick reference
TL;DR
Each scheduler iteration runs: reap finished requests and free their KV, compute the block budget, admit waiting requests, preempt under pressure, then run one batched decode step.
Picture a small ferry that crosses a river over and over. Before each crossing, the captain first lets off everyone who reached their stop, freeing up seats. Then he counts how many seats are now open. He waves new passengers aboard up to that count. If he somehow over-promised and the boat is too heavy, he asks the lowest-priority passengers to step off and wait for a later trip. Only then does he actually cross the river once, moving everyone forward by one stop. Next trip, he repeats the whole routine. The crossing itself is always the last thing, because the captain wants the fullest, safest boat before spending the fuel.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tiny single-user deployments with one request at a time There is nothing to batch, so the scheduler loop adds overhead without improving throughput.
Strict per-request latency SLAs with no tolerance for preemption Token-level admission can interleave requests and preemption adds jitter that violates tight tail latency targets.
Offline batch jobs where all inputs are known up front Static batching packs once and avoids the per-iteration admission and budgeting cost entirely.
Workloads dominated by very long prefills with little decode The loop optimizes decode density, so prefill-heavy traffic benefits more from disaggregated prefill serving.
Quick scan · 2–4 min
Everything important, quickly.
3 min: token-level scheduling motivation + the five-step order with the reason for each position + preemption as fallback + why decode is last + paged attention as the enabler.
Real-world examples
Real products, models, and research that use this idea.
1vLLM's scheduler implements exactly this iteration-level loop, freeing KV blocks on completion before recomputing the admission budget each step.
2SGLang runs continuous batching with RadixAttention prefix sharing, admitting requests against a live block budget every decode step.
3TensorRT-LLM (NVIDIA) ships in-flight batching, its name for the same per-iteration admit-and-decode loop on H100 and B200 hardware.
4Hugging Face TGI uses continuous batching with token-level scheduling so new requests join the running batch without waiting for it to drain.
5Anthropic and OpenAI serving stacks rely on continuous batching to keep decode-phase GPU utilization high across thousands of concurrent Claude Opus 4.7 and GPT-5.5 sessions.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy must reaping finished requests happen before computing the KV-block budget?
Sign in or go Pro to see the approach for this follow-up.
QWhy is preemption placed after admission rather than before it?
Sign in or go Pro to see the approach for this follow-up.
QWhat does chunked prefill change about this loop?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Putting the decode step first or admitting new requests before freeing finished ones. You must reclaim KV blocks before you can know the true budget for admission.