How does Sarathi-Serve's chunked prefill change the latency profile of a serving system?

A serving request has two phases. **Prefill** reads the whole prompt in one parallel forward pass and is compute-heavy. Decode then emits output tokens one at a time and is memory-bandwidth-heavy. When prefill and decode share a GPU, a long prompt causes a problem. A single prefill of thousands of tokens can occupy the GPU for hundreds of milliseconds. Every other request waiting for its next decode token is stuck behind it, so their per-token latency spikes. Chunked prefill, introduced by Sarathi-Serve, fixes this by slicing one long prefill into fixed-size pieces. A piece might be 512 tokens. The scheduler then processes one chunk per step instead of the whole prompt at once. Between chunks, the scheduler also runs the decode steps of other requests in the same batched forward pass. The GPU sees a hybrid batch: a prefill chunk plus a handful of decode tokens. Decode never has to wait for the entire long prompt to finish. The trade is small. The long prompt's first token arrives slightly later because its prefill is spread over several steps. In return, every other request gets steady, predictable token timing.

**Chunked prefill** splits a long prompt's prefill into fixed-size token chunks, then interleaves those chunks with ongoing decode steps inside one batched forward pass. The motivation is a scheduling conflict on co-located serving. A long prefill is one big compute job. Run it alone and it monopolizes the GPU. That stalls every concurrent decode for hundreds of milliseconds. The result is a spike in P99 time per output token, the metric users feel. Sarathi-Serve forms a hybrid batch each step. It takes one prefill chunk and packs it alongside the decode tokens of other requests. Attention masking keeps each piece correct. The decode tokens piggyback almost for free, because decode is memory-bound and the chunk's compute leaves headroom. The property gained is stall-free decode. Per-token latency stays bounded and uniform under load. The cost is modest. The chunked prompt's time to first token rises a little, since its prefill is sliced across many steps. Versus prefill-decode disaggregation, which puts the two phases on separate GPU pools, chunking needs no extra hardware. It is the pragmatic choice when you must bound tail TPOT on one shared pool.