Same topic, related formats. Practice these next.
Continuous (iteration-level) batching changes how an LLM server schedules requests compared to static batching. Describe how requests enter and leave the batch in continuous batching, and what that implies for KV-cache lifetime and allocation per request.
Continuous batching schedules at one decode step granularity, requests enter and leave each iteration, and per-request paged KV allocation frees pages as soon as a request finishes.
Imagine a bus that waits at every stop for its slowest passenger to finish reading their book before moving on. That is static batching, the whole bus crawls. Continuous batching is like a moving walkway: passengers step on when they arrive, step off the moment they reach their destination, and new passengers fill the empty space immediately. The walkway never stops for anyone. For LLM serving the walkway is the GPU running one decode step per tick, the passengers are requests, and the trick that makes 'step on and off' fast is splitting the seat space into small reusable cushions instead of giving everyone a custom-sized bench.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
7 min: static vs continuous batching contrast, iteration-level scheduling primitive, per-request KV lifetime, PagedAttention as prerequisite, composition with chunked prefill, throughput modeling, failure modes around memory pressure and preemption.
| Property | Static batching | Continuous batching |
|---|---|---|
| Scheduling granularity | Per request (batch starts and finishes together) | Per iteration (one decode step) |
| Batch composition | Fixed at batch start | Changes every iteration |
| Finished-request behavior | Slot stays idle until slowest finishes | Slot reclaimed and KV freed immediately |
| New-request admission | Wait until next batch starts | Join at the next iteration once memory available |
| KV cache lifetime | Tied to slowest request | Tied to each request individually |
| Memory layout requirement | Contiguous per-request buffers acceptable | Paged KV required to avoid fragmentation |
| Throughput on mixed-length workloads | Baseline | 5-10x higher |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Confusing continuous batching with chunked prefill. Continuous batching is request-level scheduling (which requests run this iteration); chunked prefill is prefill-side dispatching (how a long prefill gets sliced into iterations).
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.