A continuous-batching scheduler (vLLM / SGLang) supports both prefix caching and chunked prefill. Walk through what happens to a new request whose first 5,000 tokens hit the prefix cache, when the total prompt is 7,000 tokens and the chunk size is 1,024. Explain how the two optimizations stack within one scheduler iteration.
Free · 2 AI evals / day
**Prefix caching** stores the KV (key/value) state for prompt prefixes the scheduler has seen before. When a new request comes in, the scheduler checks if its prompt starts with any cached prefix. A hit means those tokens never need to be re-prefilled, the existing KV blocks are simply reattached to the new request.
In this scenario, the 7,000-token prompt has a 5,000-token cache hit. Those 5K tokens are *free* in compute terms: no forward pass, no KV computation. The scheduler attaches the cached KV blocks and moves on.
What's left is 2,000 tokens of *uncached* tail that still needs real prefill, the model must run forward through them to compute their K and V vectors and generate the first output token. This is where chunked prefill kicks in.
**Chunked prefill** says: instead of running the whole 2,000-token prefill in one giant forward pass (which would stall every other request waiting on decode), slice it into chunks of at most 1,024 tokens. Here that gives two chunks: 1,024 + 976.
In each scheduler iteration, one chunk runs *together* with the decode steps of other in-flight requests. Iteration N: chunk 1 (1,024 tokens of prefill) + decode steps for users B, C, D. Iteration N+1: chunk 2 (976 tokens) + more decodes. By iteration N+2, the new request is in pure decode mode.
The two optimizations stack: caching cut the work; chunking shaped what was left.
**The two optimizations attack different problems.** Prefix caching reduces the *amount* of prefill work. Chunked prefill reshapes whatever prefill work remains so it doesn't monopolize a scheduler iteration.
On admission, the scheduler hashes the prompt against the prefix cache (typically a radix tree of KV blocks keyed by token hashes). The 5,000-token hit means those KV blocks already exist in HBM from prior requests. The scheduler reference-counts the blocks (so they aren't evicted while this request uses them) and attaches them to the new request as the prefix of its KV cache. Zero compute. The hit region's contribution to TTFT is essentially the cost of the hash lookup and block-reference bookkeeping, milliseconds.
The remaining 2,000-token tail is genuine prefill work. The model must run forward through tokens 5,001-7,000, attending over the entire 7,000-token KV state (including the cached portion), computing K/V for the new tokens, and generating logits for token 7,001.
Chunked prefill slices this tail under the 1,024-token chunk size. Two chunks result: tokens 5,001-6,024 (1,024 tokens) and 6,025-7,000 (976 tokens). Each chunk is its own forward pass scheduled inside one continuous-batching iteration *alongside* the decode steps of other in-flight requests. In iteration t the GPU executes: chunk 1 of this prefill + decode-step-t for each of the other N requests, all in one batched forward pass.
The stacking effect is clean. Prefix cache: work reduction. Chunked prefill: shape smoothing. TTFT for the new request drops sharply because most of the prompt was free. Decode latency for other requests stays bounded because no single prefill monopolizes the GPU.
Quick reference
TL;DR
Prefix cache reuses 5K tokens of existing KV, that prefill is skipped entirely.
Picture a teacher who needs to read a 7-page essay before grading. With prefix caching, she realizes she already read the first 5 pages last week and has notes, she just attaches those notes and starts fresh on page 6. Now she has 2 pages to read. Chunked prefill is her policy of never sitting down for a long uninterrupted read. She reads roughly a page (1,024 tokens), then briefly checks in with five other students' work in progress, then reads the rest of the second page, then checks in again. Every student gets a steady stream of attention instead of waiting an hour while she finishes one essay.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Workloads with no prefix overlap across requests Prefix caching has nothing to hit, pure cache miss. Chunked prefill alone still helps with head-of-line blocking from long prompts.
Single-request serving with no concurrency Chunked prefill's benefit is interleaving with other requests' decode steps; with one request there's no interleaving to gain.
Very short prompts (<1 chunk) and short generations Both optimizations target longer-prompt regimes; chunk size is already larger than the prompt, and prefix-cache lookups add small overhead.
Strict per-request determinism requirements Continuous batching's interleaving and prefix caching's KV reuse can introduce subtle numerical non-determinism vs single-request serving; some safety-critical applications cannot tolerate it.
Quick scan · 2–4 min
Everything important, quickly.
3 min: prefix caching as KV-block reuse + chunked prefill as head-of-line-blocking fix + the specific 5K/2K/1024 walkthrough + how they batch into iterations + the orthogonality argument.
Real-world examples
Real products, models, and research that use this idea.
1SGLang's RadixAttention combines prefix caching with continuous batching and is the canonical 2026 reference for this composition.
2vLLM 0.6+ supports automatic prefix caching with `--enable-prefix-caching` and chunked prefill via `--enable-chunked-prefill`; teams often run both together by default.
3Anthropic's prompt caching feature is the API-level analog of prefix caching, exposed to customers as billable cache reads at 0.1x rate.
4TensorRT-LLM's KV reuse feature plays the same role as prefix caching on NVIDIA's serving stack.
5Sarathi-Serve (Agrawal et al., 2024) is the academic origin of chunked prefill; vLLM and SGLang adopted the technique in production.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the scheduler decide which prefix-cache blocks to evict when HBM pressure rises?
Sign in or go Pro to see the approach for this follow-up.
QWhat's the right chunk size on an H100 running Llama 3.1 70B?
Sign in or go Pro to see the approach for this follow-up.
QWhy is prefix caching not strictly a win for unique per user prompts?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing the cache hit just speeds up prefill, it skips it entirely for the hit region. Or believing chunked prefill helps the cached portion, it doesn't; it only shapes the remaining work.