How does Sarathi-Serve's chunked prefill change the latency profile of a serving system?

Attempt it

Explain Sarathi-Serve's chunked prefill: what it does, what it solves on a co-located prefill/decode system, and what it costs.

Free · 2 AI evals / day

Quick reference

TL;DR

Chunked prefill slices a long prompt into fixed-size pieces and interleaves them with running decodes in one batch, so decode never stalls; the cost is slightly higher TTFT for the chunked prompt.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a kitchen with one chef and one stove. A huge banquet order (a long prompt) arrives. If the chef cooks the whole banquet in one uninterrupted block, every other diner waiting for one more bite of their meal sits hungry for minutes. Chunked prefill tells the chef to cook the banquet in small batches, and between each batch, plate one more bite for every waiting diner. The banquet finishes a touch later overall, because it was sliced up. But nobody else ever waits a long, unpredictable time for their next bite. The stove stays busy the whole time, and everyone gets steady, predictable service instead of one person hogging the kitchen.

Key concepts

Chunked prefill Prefill vs decode Continuous batching TPOT and TTFT

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: prefill stalls decode on a shared pool, chunk the prefill, piggyback decodes into a hybrid batch, stall-free decode, the TTFT versus TPOT trade, and chunking versus disaggregation.

Real-world examples

Real products, models, and research that use this idea.

vLLM ships chunked prefill as a scheduler option, mixing prefill chunks with decode tokens under a single per-step token budget.
TensorRT-LLM (NVIDIA) implements chunked prefill as its production reference for bounding decode latency on H100 and B200.
SGLang interleaves prefill chunks with running decodes to keep per-token latency stable when serving Llama 4 and Qwen 3 under load.
DistServe and Mooncake take the opposite path, disaggregating prefill and decode onto separate GPU pools instead of chunking on one.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can decode tokens piggyback onto a prefill chunk so cheaply?

Sign in or go Pro to see the approach for this follow-up.

QHow do you choose the chunk size or per-step token budget?

Sign in or go Pro to see the approach for this follow-up.

QWhen does disaggregation beat chunked prefill?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying chunked prefill speeds up the prefill itself. It does not. It slices prefill so decode is not starved, trading a little extra TTFT for far steadier per-token latency.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve