What problem does chunked prefill solve in production LLM serving?

MCQ·Medium·4.0 · 0·~1 min·Asked atAlibaba Mistral AI·Relevant atAi4bharat Cerebras Deepseek NVIDIA

Attempt it

**The answer is B.** In LLM serving there are two distinct workloads happening on the same GPU: - **Prefill**: processing a user's prompt. Many query positions at once, compute-bound, makes great use of GPU FLOPs. - **Decode**: generating one token at a time. One query position per step, memory bandwidth bound (the KV cache is the bottleneck), poor FLOP utilization. Without chunked prefill the scheduler is stuck. If a new user arrives with a 10000-token prompt, the prefill alone takes hundreds of milliseconds. During that time, every other request in the batch that is in decode phase is blocked. Decode latency spikes; the tail-latency SLO breaks. **Chunked prefill** is the scheduler's fix. The long prefill is broken into chunks (say 512 tokens each). At every scheduler step, the scheduler can mix: one chunk of the new prefill PLUS decode steps from all the other active requests. The GPU stays saturated and decode latency stays low. Why the other options are wrong: - **A** describes sparse attention (sliding window, BigBird), it changes the FLOP count, not the schedule. - **C** describes PagedAttention, a memory-layout primitive. - **D** describes KV-cache quantization, a precision optimization. All three are real, separate optimizations. Chunked prefill is purely scheduling.

**B is correct.** Prefill of a long prompt is one large compute-bound operation. On its own it starves co-running decode requests, which are memory bandwidth bound and prefer interleaved scheduling. Run a 10k-token prefill end to end and every decode user in the batch eats the full prefill latency as added wait. **Chunked prefill** splits the prefill into smaller pieces (typically 512 or 1024 tokens) that the scheduler can interleave with decode steps in the same batch. At every iteration: - Compute-bound work from the prefill chunk uses GPU FLOPs. - Memory-bound work from concurrent decode steps uses GPU bandwidth. The two workloads complement each other on the hardware: tensor cores stay busy while HBM bandwidth feeds decode reads. The GPU never flips between underutilized decode-only and oversubscribed prefill phases. **Why the distractors fail:** - A confuses chunked prefill with sparse attention. Chunked prefill does not change the FLOP count, just when chunks run. - C describes PagedAttention, the paged KV cache abstraction, orthogonal. - D describes KV-cache quantization, a precision-level optimization, also orthogonal. Both vLLM and TGI ship chunked prefill as a default scheduler option in 2026 production deployments.

TL;DR

Chunked prefill splits a long prompt's prefill into scheduler-sized pieces so they can interleave with decode steps from other requests in the same batch.

Memory aid

Easy to grasp

Picture a restaurant kitchen where one chef just got a 20-step gourmet order and the other diners are waiting for their drinks refilled. If the chef finishes the entire 20-step meal before pouring anyone a glass of water, the other tables get angry. Chunked prefill is like the rule that says: do two steps of the big meal, then refill some drinks, then two more steps, then more drinks. The big meal still gets done; the other tables stop waiting. In LLM serving the big meal is a long prompt's prefill and the drink refills are quick decode steps from other users.

Key concepts

Chunked prefill Prefill phase Continuous batching Decode phase

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Chunked prefill is one of the four pillars of the 2026 LLM-serving stack, alongside PagedAttention, continuous batching, and prefix sharing. It is a scheduler-side mechanism that lets a long prompt's prefill be processed in pieces, interleaved with decode steps from other requests in the same batch.

The motivating insight is that prefill and decode are different workload regimes on the same GPU. Prefill is compute-bound; decode is memory bandwidth bound. Co-running them in the same batch keeps the GPU saturated across both hardware axes. But a long indivisible prefill blocks the interleave promise, so the scheduler needs the ability to slice prefill work into chunks.

This deep dive walks through the two regimes, why naive scheduling stalls, how chunking enables hybrid scheduler iterations, what knobs operators tune, and how chunked prefill composes with the other production cache primitives.

Prefill and decode are different workloads

LLM inference has two distinct phases. Understanding their hardware behavior is the prerequisite for understanding why chunked prefill exists.

Prefill phase

The model processes the user's full prompt in one forward pass. For a 2000-token prompt:

Q has shape (2000, d), K has shape (2000, d), V has shape (2000, d) per layer per head.
Attention is O(n^2 d) per layer, which for n = 2000 saturates tensor cores.
Arithmetic intensity is ~200 ops/byte, well above the H100 balance point (~10 ops/byte for fp16). Kernel is compute-bound.

Decode phase

The model generates one new token at a time. For a request at position seq_len = 2000:

Q has shape (1, d) per layer per head, K and V are the full cached (2000, d) per layer per head.
Attention is O(n d) per layer, which is tiny FLOPs.
Arithmetic intensity is ~1 op/byte, well below the balance point. Kernel is memory bandwidth bound. Tensor cores idle while HBM streams the cached K, V.

Why this matters for scheduling

The GPU has two underutilized hardware axes in different regimes. Prefill saturates tensor cores but leaves bandwidth slack. Decode saturates bandwidth but leaves tensor cores slack. A scheduler that runs one phase at a time wastes 50% of the GPU half the time. A scheduler that mixes phases saturates both axes at once.

Why naive scheduling stalls

Hybrid scheduler iterations: what runs in one step

Tuning chunk size and trade-offs

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	Without chunked prefill	With chunked prefill
Long-prefill behavior	Blocks all decode users in the batch	Interleaves with decode at chunk granularity
GPU utilization	Alternates compute-bound and bandwidth-bound phases	Hybrid: tensor cores + HBM bandwidth saturated simultaneously
Tail decode latency	Spikes to length of longest prefill	Bounded by chunk_size processing time
Prefill TTFT for the prefilling request	Lower (no chunk overhead)	Slightly higher (chunk boundary cost)
What it requires	Standard scheduler	Paged KV + mixed-batch attention kernel

Real products, models, and research that use this idea.

vLLM ships chunked prefill as a default scheduler option, used in production deployments serving Llama 4 Maverick and Qwen 3.5 long-context workloads.
TGI (Hugging Face Text Generation Inference) added chunked prefill in 2024, now standard for serving Gemma 4 and DeepSeek V4 with mixed prefill-decode batches.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does chunk_size interact with the GPU's compute to bandwidth balance?

The chunk should be large enough that the prefill work in one scheduler step saturates tensor cores, but small enough that the bandwidth share left for co-running decode steps is sufficient. On H100, prefill chunks of 512-2048 tokens typically hit the sweet spot. Going below ~128 starts to underutilize tensor cores per chunk; going above ~4096 starts to crowd out decode bandwidth and decode latency creeps up.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing chunked prefill with PagedAttention or with sparse attention. Chunked prefill is scheduler-side: it changes when prefill work happens, not the FLOP count or the memory layout.

60 second bullets to scan on the way to the call.

Difference between prefill (compute-bound) and decode (bandwidth-bound) regimes
Why a long prefill blocks co-running decode users without chunking

Primary sources. Browse if you want the original framing.