Walk through how a prefix cache hit changes the work that chunked prefill has to do

**Prefix caching** stores the KV (key/value) state for prompt prefixes the scheduler has seen before. When a new request comes in, the scheduler checks if its prompt starts with any cached prefix. A hit means those tokens never need to be re-prefilled, the existing KV blocks are simply reattached to the new request. In this scenario, the 7,000-token prompt has a 5,000-token cache hit. Those 5K tokens are *free* in compute terms: no forward pass, no KV computation. The scheduler attaches the cached KV blocks and moves on. What's left is 2,000 tokens of *uncached* tail that still needs real prefill, the model must run forward through them to compute their K and V vectors and generate the first output token. This is where chunked prefill kicks in. **Chunked prefill** says: instead of running the whole 2,000-token prefill in one giant forward pass (which would stall every other request waiting on decode), slice it into chunks of at most 1,024 tokens. Here that gives two chunks: 1,024 + 976. In each scheduler iteration, one chunk runs *together* with the decode steps of other in-flight requests. Iteration N: chunk 1 (1,024 tokens of prefill) + decode steps for users B, C, D. Iteration N+1: chunk 2 (976 tokens) + more decodes. By iteration N+2, the new request is in pure decode mode. The two optimizations stack: caching cut the work; chunking shaped what was left.

**The two optimizations attack different problems.** Prefix caching reduces the *amount* of prefill work. Chunked prefill reshapes whatever prefill work remains so it doesn't monopolize a scheduler iteration. On admission, the scheduler hashes the prompt against the prefix cache (typically a radix tree of KV blocks keyed by token hashes). The 5,000-token hit means those KV blocks already exist in HBM from prior requests. The scheduler reference-counts the blocks (so they aren't evicted while this request uses them) and attaches them to the new request as the prefix of its KV cache. Zero compute. The hit region's contribution to TTFT is essentially the cost of the hash lookup and block-reference bookkeeping, milliseconds. The remaining 2,000-token tail is genuine prefill work. The model must run forward through tokens 5,001-7,000, attending over the entire 7,000-token KV state (including the cached portion), computing K/V for the new tokens, and generating logits for token 7,001. Chunked prefill slices this tail under the 1,024-token chunk size. Two chunks result: tokens 5,001-6,024 (1,024 tokens) and 6,025-7,000 (976 tokens). Each chunk is its own forward pass scheduled inside one continuous-batching iteration *alongside* the decode steps of other in-flight requests. In iteration t the GPU executes: chunk 1 of this prefill + decode-step-t for each of the other N requests, all in one batched forward pass. The stacking effect is clean. Prefix cache: work reduction. Chunked prefill: shape smoothing. TTFT for the new request drops sharply because most of the prompt was free. Decode latency for other requests stays bounded because no single prefill monopolizes the GPU.