What problem does chunked prefill solve in production LLM serving?

**The answer is B.** In LLM serving there are two distinct workloads happening on the same GPU: - **Prefill**: processing a user's prompt. Many query positions at once, compute-bound, makes great use of GPU FLOPs. - **Decode**: generating one token at a time. One query position per step, memory bandwidth bound (the KV cache is the bottleneck), poor FLOP utilization. Without chunked prefill the scheduler is stuck. If a new user arrives with a 10000-token prompt, the prefill alone takes hundreds of milliseconds. During that time, every other request in the batch that is in decode phase is blocked. Decode latency spikes; the tail-latency SLO breaks. **Chunked prefill** is the scheduler's fix. The long prefill is broken into chunks (say 512 tokens each). At every scheduler step, the scheduler can mix: one chunk of the new prefill PLUS decode steps from all the other active requests. The GPU stays saturated and decode latency stays low. Why the other options are wrong: - **A** describes sparse attention (sliding window, BigBird), it changes the FLOP count, not the schedule. - **C** describes PagedAttention, a memory-layout primitive. - **D** describes KV-cache quantization, a precision optimization. All three are real, separate optimizations. Chunked prefill is purely scheduling.

**B is correct.** Prefill of a long prompt is one large compute-bound operation. On its own it starves co-running decode requests, which are memory bandwidth bound and prefer interleaved scheduling. Run a 10k-token prefill end to end and every decode user in the batch eats the full prefill latency as added wait. **Chunked prefill** splits the prefill into smaller pieces (typically 512 or 1024 tokens) that the scheduler can interleave with decode steps in the same batch. At every iteration: - Compute-bound work from the prefill chunk uses GPU FLOPs. - Memory-bound work from concurrent decode steps uses GPU bandwidth. The two workloads complement each other on the hardware: tensor cores stay busy while HBM bandwidth feeds decode reads. The GPU never flips between underutilized decode-only and oversubscribed prefill phases. **Why the distractors fail:** - A confuses chunked prefill with sparse attention. Chunked prefill does not change the FLOP count, just when chunks run. - C describes PagedAttention, the paged KV cache abstraction, orthogonal. - D describes KV-cache quantization, a precision-level optimization, also orthogonal. Both vLLM and TGI ship chunked prefill as a default scheduler option in 2026 production deployments.