**The answer is B.**
In LLM serving there are two distinct workloads happening on the same GPU:
- **Prefill**: processing a user's prompt. Many query positions at once, compute-bound, makes great use of GPU FLOPs.
- **Decode**: generating one token at a time. One query position per step, memory bandwidth bound (the KV cache is the bottleneck), poor FLOP utilization.
Without chunked prefill the scheduler is stuck. If a new user arrives with a 10000-token prompt, the prefill alone takes hundreds of milliseconds. During that time, every other request in the batch that is in decode phase is blocked. Decode latency spikes; the tail-latency SLO breaks.
**Chunked prefill** is the scheduler's fix. The long prefill is broken into chunks (say 512 tokens each). At every scheduler step, the scheduler can mix: one chunk of the new prefill PLUS decode steps from all the other active requests. The GPU stays saturated and decode latency stays low.
Why the other options are wrong:
- **A** describes sparse attention (sliding window, BigBird), it changes the FLOP count, not the schedule.
- **C** describes PagedAttention, a memory-layout primitive.
- **D** describes KV-cache quantization, a precision optimization.
All three are real, separate optimizations. Chunked prefill is purely scheduling.
**B is correct.**
Prefill of a long prompt is one large compute-bound operation. On its own it starves co-running decode requests, which are memory bandwidth bound and prefer interleaved scheduling. Run a 10k-token prefill end to end and every decode user in the batch eats the full prefill latency as added wait.
**Chunked prefill** splits the prefill into smaller pieces (typically 512 or 1024 tokens) that the scheduler can interleave with decode steps in the same batch. At every iteration:
- Compute-bound work from the prefill chunk uses GPU FLOPs.
- Memory-bound work from concurrent decode steps uses GPU bandwidth.
The two workloads complement each other on the hardware: tensor cores stay busy while HBM bandwidth feeds decode reads. The GPU never flips between underutilized decode-only and oversubscribed prefill phases.
**Why the distractors fail:**
- A confuses chunked prefill with sparse attention. Chunked prefill does not change the FLOP count, just when chunks run.
- C describes PagedAttention, the paged KV cache abstraction, orthogonal.
- D describes KV-cache quantization, a precision-level optimization, also orthogonal.
Both vLLM and TGI ship chunked prefill as a default scheduler option in 2026 production deployments.
Quick reference
TL;DR
Chunked prefill splits a long prompt's prefill into scheduler-sized pieces so they can interleave with decode steps from other requests in the same batch.
Picture a restaurant kitchen where one chef just got a 20-step gourmet order and the other diners are waiting for their drinks refilled. If the chef finishes the entire 20-step meal before pouring anyone a glass of water, the other tables get angry. Chunked prefill is like the rule that says: do two steps of the big meal, then refill some drinks, then two more steps, then more drinks. The big meal still gets done; the other tables stop waiting. In LLM serving the big meal is a long prompt's prefill and the drink refills are quick decode steps from other users.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-request inference or batch size one deployments With no co-running decode users to benefit, chunking just adds scheduling overhead and slightly raises TTFT for the only active request.
All-prefill workloads like embedding generation If every request is just one forward pass with no decode phase, there is nothing to interleave with, the scheduler benefit disappears.
Very short prompts under the chunk size Prompts shorter than chunk_size complete in one iteration anyway, chunking is a no-op and the bookkeeping is pure overhead.
Latency-critical real-time TTFT targets for the prefilling request itself Chunking adds chunk-boundary overhead to the prefill, so a prefill that needed to finish in 100ms exactly will be a few percent slower. The win is for OTHER requests, not the prefilling one.
Quick scan · 2–4 min
Everything important, quickly.
6 min: prefill vs decode regimes, why naive scheduling stalls, how chunking enables hybrid scheduler iterations, chunk_size tuning, TTFT vs ITL trade-off, mixed-batch kernel requirement, distinction from PagedAttention and KV quantization.
Side-by-side
Property
Without chunked prefill
With chunked prefill
Long-prefill behavior
Blocks all decode users in the batch
Interleaves with decode at chunk granularity
GPU utilization
Alternates compute-bound and bandwidth-bound phases
Real products, models, and research that use this idea.
1vLLM ships chunked prefill as a default scheduler option, used in production deployments serving Llama 4 Maverick and Qwen 3.5 long-context workloads.
2TGI (Hugging Face Text Generation Inference) added chunked prefill in 2024, now standard for serving Gemma 4 and DeepSeek V4 with mixed prefill-decode batches.
3SGLang combines chunked prefill with RadixAttention prefix caching to serve agent workloads on Claude Opus 4.7 style long system prompt patterns.
4TensorRT-LLM exposes chunked prefill in its mixed-batch scheduler for enterprise GPT-5.5 deployments on Hopper and Blackwell GPUs.
5Sarathi-Serve introduced the formal study of chunked prefill scheduling, showing 2-3x throughput improvements on mixed prefill-decode workloads.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does chunk_size interact with the GPU's compute to bandwidth balance?
Sign in or go Pro to see the approach for this follow-up.
QWhy does chunked prefill need PagedAttention as a prerequisite?
Sign in or go Pro to see the approach for this follow-up.
QHow does chunked prefill change the prefill latency itself?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing chunked prefill with PagedAttention or with sparse attention. Chunked prefill is scheduler-side: it changes when prefill work happens, not the FLOP count or the memory layout.