- 1Reap any requests that finished generation in the previous step (EOS, stop sequence, or max_tokens hit) and free their KV blocks
- 2Run one batched decode step over the resulting set of active requests, advancing each by exactly one token
- 3Admit waiting requests up to the remaining budget if nothing was preempted, performing prefill (or chunked prefill) for newly admitted ones
- 4If running requests cannot get KV blocks for their next tokens, preempt the lowest priority running requests, freeing their KV blocks so they are recomputed later
- 5Compute the available KV block budget given current GPU memory and any in flight reservations
Each scheduler iteration runs: reap finished requests and free their KV, compute the block budget, preempt running requests that cannot grow, admit waiting requests, then run one batched decode step.
Picture a small ferry that crosses a river over and over. Before each crossing, the captain first lets off everyone who reached their stop, freeing up seats. Then he counts how many seats are now open. Passengers already aboard need a little more room each trip as their bags grow. If there is not enough, he asks the lowest-priority passengers to step off and wait for a later trip. Only if nobody had to step off does he wave new passengers aboard, up to the open seats. Only then does he cross the river once, moving everyone forward by one stop. Next trip, he repeats the whole routine. The crossing itself is always the last thing, because the captain wants the fullest, safest boat before spending the fuel.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Continuous batching is the single largest throughput lever in LLM serving, and a scheduler loop drives it. The naive alternative is static batching. It groups a fixed set of requests, runs them together, and waits for the slowest one to finish before starting the next group. That wastes the GPU badly: short requests finish early and their slots sit idle while a long request keeps running.
Continuous batching, sometimes called in-flight batching, instead rebuilds the active batch on every decode step. Requests join and leave mid-flight. The scheduler runs a tight loop, and within each iteration the steps follow a strict order that is not arbitrary. The order encodes a resource invariant. You free memory before you measure it, and you serve running requests before new ones. You run the expensive forward pass only once the batch is as dense as the memory ceiling legally allows.
Paged attention is the primitive that makes block-level allocation cheap enough for this loop to run on every step.
Why token-level scheduling beats whole-request batching
In a transformer decode loop the GPU is memory-bandwidth-bound, not compute-bound. Each step streams the model weights, plus each sequence's KV cache, from HBM to compute one new token per sequence. The marginal cost of adding another sequence to that step is small, because the weight read is shared across the batch. This is why batching is the biggest single throughput lever in inference.
Static batching fails to exploit this. It admits a group, runs it to completion, and leaves freed slots idle until the whole group drains. A batch with one 2000-token generation and seven 50-token generations wastes seven slots for most of its life.
Continuous batching fixes the idle-slot problem by scheduling at the granularity of a single token. After every decode step the batch is reconstituted. Finished sequences leave, waiting sequences enter, and the GPU stays densely packed. The cost of this flexibility is a scheduler that must run a careful bookkeeping routine before every forward pass.
That routine is the five-step iteration. It is a memory manager that runs before each batched forward pass. The scheduler does not run the matmul itself. It decides which sequences belong in the next batch, allocates and frees their KV blocks, and then hands the assembled batch to the model. Because this happens every token, the bookkeeping must be cheap. Paged allocation provides that. The throughput payoff is large. Continuous batching commonly delivers several times the tokens per second of static batching on the same hardware, almost entirely by removing idle slots.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM's scheduler implements exactly this iteration-level loop, freeing KV blocks on completion before recomputing the admission budget each step.
- SGLang runs continuous batching with RadixAttention prefix sharing, admitting requests against a live block budget every decode step.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy must reaping finished requests happen before computing the KV-block budget?
A finished sequence keeps holding its KV blocks until reaped. Budgeting before reaping reads stale occupancy, so you under-admit and waste capacity. Free first, then the measured free-block count is accurate, then admit against it.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Putting the decode step first or admitting new requests before freeing finished ones. You must reclaim KV blocks before you can know the true budget for admission.
60 second bullets to scan on the way to the call.
Why finished requests must be reclaimed before the budget is measured
What the KV-block budget depends on and how reservations affect it
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.