Describe how continuous batching interacts with the KV cache

Short answer·Medium·4.0 · 0·~3 min·Asked atFigure Ai Kpmg Mphasis·Relevant atAi4bharat Anthropic Cerebras Deepseek

Attempt it

Continuous (iteration level) batching changes how an LLM server schedules requests compared to static batching. Describe how requests enter and leave the batch in continuous batching, and what that implies for KV-cache lifetime and allocation per request.

Free · 2 AI evals / day

**Continuous batching** is the request-level scheduling pattern that powers every modern LLM serving stack. ### Static batching (the old way) A fixed set of N requests starts together. Every decode iteration runs the model on all N in lockstep. The batch finishes when the slowest request hits EOS or max tokens. Until then, slots for requests that finished early sit idle, consuming GPU memory but producing nothing. KV-cache lifetime: tied to the slowest request. If 7 requests finish at 50 tokens but one runs to 2000, the cache for all 7 stays alive (or gets wastefully shifted) until the long one finishes. Memory utilization is awful. ### Continuous batching (the modern way) The scheduler picks the active batch fresh at every iteration. Finished requests leave immediately and free their KV. Newly arrived requests can join at the next iteration. The batch composition changes step by step. KV-cache lifetime: tied to each individual request, not to the batch. A request's KV is allocated when it joins, written each decode step, and freed the instant it finishes. ### Why PagedAttention is the natural pairing If every request has its own KV with independent lifetime, you need cheap per-request allocation and freeing. PagedAttention's paged allocation does exactly that: fixed-size physical blocks in a pool, allocate on demand per request, free immediately on request completion. Without paging you would need contiguous per-request buffers, which fragment terribly under continuous batching. ### Throughput win The GPU never idles waiting for the slowest sequence. Short requests free their pages quickly, long requests keep streaming, the scheduler keeps the batch as full as memory allows. Aggregate throughput is 5-10x higher than static batching on realistic mixed-length workloads.

In **static (request-level) batching**, a fixed set of N requests starts together, runs decode steps in lockstep, and finishes together. The slowest request pins the batch. The KV cache for the whole batch lives for the duration of the longest sequence, leaving GPU memory occupied by finished sequences and idle slots. **Continuous batching** (also called iteration-level batching, popularized by Orca and shipped in vLLM, TGI, TensorRT-LLM) schedules at the iteration granularity. At every decode step the scheduler picks the set of active requests; finished requests leave the batch immediately and freshly arrived requests can join the next step. The batch composition changes step by step. **For the cache, each request owns its own KV state.** When a request finishes (hits EOS or max tokens), its KV pages are freed right then and become available for the next request to allocate. **Paged KV (PagedAttention) is the natural pairing**: the cache is split into fixed-size pages with a per-request page table, so allocating and freeing per request is cheap and avoids the fragmentation a contiguous per-request buffer would cause. The scheduler maintains a free pool of physical blocks; admit a request when free_blocks >= initial_need; release blocks the moment the request completes. **Throughput rises** because the GPU never idles waiting for the slowest sequence in a fixed batch. Short requests free their pages quickly, long requests keep streaming, and the scheduler keeps the per-step batch as full as memory allows. Production deployments report 5-10x throughput improvements on mixed-length workloads.

TL;DR

Continuous batching schedules at one decode step granularity, requests enter and leave each iteration, and per-request paged KV allocation frees pages as soon as a request finishes.

Memory aid

Easy to grasp

Imagine a bus that waits at every stop for its slowest passenger to finish reading their book before moving on. That is static batching, the whole bus crawls. Continuous batching is like a moving walkway: passengers step on when they arrive, step off the moment they reach their destination, and new passengers fill the empty space immediately. The walkway never stops for anyone. For LLM serving the walkway is the GPU running one decode step per tick, the passengers are requests, and the trick that makes 'step on and off' fast is splitting the seat space into small reusable cushions instead of giving everyone a custom-sized bench.

Key concepts

Continuous batching KV cache PagedAttention Static batching

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Continuous batching is the request-level scheduling primitive that, together with PagedAttention and chunked prefill, powers every major 2026 LLM serving stack (vLLM, TGI, TensorRT-LLM, SGLang, LMDeploy). It replaces the older static (request-level) batching model with iteration-level scheduling, dramatically improving GPU utilization on realistic mixed-length workloads.

This deep dive walks the difference between the two scheduling models, explains how requests enter and leave the batch each iteration, traces the KV-cache lifetime implication, and ties everything back to PagedAttention as the prerequisite memory primitive that makes continuous batching practical. It also covers how continuous batching composes with the other scheduling mechanisms (chunked prefill, speculative decoding, prefix sharing) in the modern production stack.

Static batching: what fails and why

The original transformer serving model is static batching: collect N requests into a batch, run them through prefill and decode in lockstep, finish when the slowest one finishes.

How it works

A batch of N requests is constructed (often N is fixed by serving framework).
All N run prefill together (often padded to the max prefill length).
Decode runs in lockstep: every iteration produces one new token for every active request.
The batch terminates when the slowest request hits EOS or max_new_tokens.

Three failure modes

Head of line blocking. New requests cannot join until the current batch finishes. If your batch has one 2000-token-output request and 31 short ones, every new arrival waits for that 2000-token request to drain.

Slot waste from early-finishing requests. Requests that hit EOS at iteration 50 leave their slot effectively idle for the remaining 1950 iterations (in the worst case). The GPU still processes the batched matmul for those slots, but the outputs are discarded. KV memory stays allocated.

Memory waste from padding. Prefill is often padded to the longest prompt in the batch. A batch with one 4000-token prompt and 31 100-token prompts pays 4000 tokens of prefill compute per request.

Quantitative impact

On realistic mixed-length workloads (some short, some long, modeled as a heavy-tailed length distribution), static batching utilizes maybe 20-30% of the theoretical GPU throughput. The other 70-80% is wasted on idle slots, padded prefill, and head of line blocking.

Continuous batching: the iteration-level scheduling model

Per-request KV cache lifetime

Why PagedAttention is the prerequisite

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	Static batching	Continuous batching
Scheduling granularity	Per request (batch starts and finishes together)	Per iteration (one decode step)
Batch composition	Fixed at batch start	Changes every iteration
Finished-request behavior	Slot stays idle until slowest finishes	Slot reclaimed and KV freed immediately
New-request admission	Wait until next batch starts	Join at the next iteration once memory available
KV cache lifetime	Tied to slowest request	Tied to each request individually
Memory layout requirement	Contiguous per-request buffers acceptable	Paged KV required to avoid fragmentation
Throughput on mixed-length workloads	Baseline	5-10x higher

Real products, models, and research that use this idea.

vLLM is the canonical 2026 production stack shipping continuous batching plus PagedAttention plus chunked prefill, used to serve Llama 4 Maverick, Qwen 3.5, and DeepSeek V4.
Orca (OSDI 2022) introduced iteration-level batching; vLLM productionized it with PagedAttention as the cache primitive.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does continuous batching interact with speculative decoding?

Speculative decoding generates K candidate tokens with a smaller draft model and verifies them with the target model in one forward pass. Per request, the iteration produces K tokens (on accept) or 1 (on first reject). Continuous batching handles this naturally: each request's per-iteration KV growth is variable (1 to K blocks worth), the scheduler just tracks per-request progress. The two compose orthogonally, scheduling at the iteration level, generating multiple tokens per iteration.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing continuous batching with chunked prefill. Continuous batching is request-level scheduling (which requests run this iteration); chunked prefill is prefill-side dispatching (how a long prefill gets sliced into iterations).

60 second bullets to scan on the way to the call.

Difference between request-level (static) and iteration-level (continuous) scheduling
Why static batching wastes GPU time waiting for slowest sequence

Primary sources. Browse if you want the original framing.