How a prefix cache hit changes the work that chunked prefill has to do

**Prefix caching** stores the KV (key/value) state of prompt prefixes the server has seen before. When a new request arrives, the scheduler checks whether its prompt starts with a cached prefix. On a hit, those tokens need no prefill: the existing KV blocks are reattached to the new request. Here, the 7,000-token prompt has a 5,000-token hit. Those 5,000 tokens cost no forward pass and no KV computation. The 2,000-token *uncached* tail still needs real prefill: the model runs forward through it to compute its K and V vectors and the first output token. **Chunked prefill** splits that work into chunks of at most 1,024 tokens, instead of one large forward pass that would stall other requests' decode. Here that gives two chunks: 1,024 + 976. In each scheduler iteration, one chunk runs *together* with the decode steps of other in-flight requests. Iteration N: chunk 1 plus decode steps for users B, C, D. Iteration N+1: chunk 2 plus more decodes. From iteration N+2 the new request is decoding. The two optimizations stack: caching cut the work; chunking split what was left.

**The two optimizations solve different problems.** Prefix caching reduces the *amount* of prefill work. Chunked prefill splits whatever prefill work remains so it doesn't take over a scheduler iteration. On admission, the scheduler matches the prompt against the prefix cache (vLLM hashes full 16-token KV blocks; SGLang walks a radix tree token by token). The 5,000-token hit means those KV blocks already exist in HBM. The scheduler reference-counts the blocks (so they aren't evicted while this request uses them) and attaches them as the prefix of the new request's KV cache. No forward pass runs for those tokens; the cost is lookup and bookkeeping. The remaining 2,000-token tail is real prefill work. The model runs forward through tokens 5,001-7,000, attending over the full 7,000-token KV state (including the cached part), computing K/V for the new tokens, and producing logits for token 7,001. Chunked prefill splits this tail under the 1,024-token chunk size into two chunks: tokens 5,001-6,024 (1,024 tokens) and 6,025-7,000 (976 tokens). Each chunk runs in one continuous-batching iteration *alongside* the decode steps of other in-flight requests: in iteration t, chunk 1 plus one decode step for each of the other N requests, all in one batched forward pass. Prefix caching cuts the work; chunked prefill splits what remains. TTFT for the new request drops sharply because most of the prompt needs no compute. Decode latency for other requests stays bounded because no single prefill takes over the GPU.