Identify the three conditions under which disaggregated prefill and decode (DistServe-style architectures) deliver a real win over a co-located continuous-batching deployment. For each condition, explain what fails when it is absent.
Free · 2 AI evals / day
**Prefill and decode are very different workloads.** Prefill processes the entire prompt in one big parallel pass and is compute bound. Decode generates one token at a time, reads the whole KV cache from HBM every step, and is memory bandwidth bound. Running both on the same GPU means a long prefill can block several decode steps and add latency to other in-flight requests.
Disaggregated serving (DistServe, Splitwise, and similar 2024-2026 architectures) splits these into two replica pools. The prefill pool handles new prompts and ships the resulting KV cache to the decode pool over a fast interconnect. The decode pool handles ongoing generation. Each pool can be sized, scheduled, and tuned independently for its own bottleneck.
The win shows up only when three conditions hold together. First, prompts need to be long enough that prefill is a meaningful share of total time. Short prompts make prefill a thin slice that does not interfere with decode anyway. Second, traffic has to be heavy enough to keep both pools saturated; otherwise one sits idle while the other queues, and your effective capacity is worse than a flexible co-located stack. Third, the interconnect between pools must be fast enough that shipping the KV cache costs less than the latency it saves.
When all three hold, disaggregation cuts tail latency for both phases. When any one fails, a single co-located stack running continuous batching with chunked prefill usually wins on simplicity, utilization, and cost.
**Disaggregation is a scale-driven optimization, not a default architecture.** It splits the two phases of LLM serving onto separate replica pools because prefill (compute bound, big parallel matmul) and decode (memory bound, KV-cache sweep) have different bottlenecks and interfere with each other when co-located.
Three conditions have to hold together. **Prompts long enough.** Long-context RAG, long system prompts, multi-turn chats. Short prompts make prefill a tiny slice; isolating it costs more than it saves. **Traffic high enough.** Both pools have to stay busy on their own work. At modest QPS one pool sits idle while the other queues, and the co-located stack with flexible scheduling beats the disaggregated one on effective capacity. **Interconnect fast enough.** The KV cache for the prefilled prompt has to ship from the prefill pool to the decode pool. Over NVLink-Switch or RDMA at hundreds of GB/s the transfer is invisible. Over slower fabric, you spend a meaningful fraction of a decode step shipping bytes, and the latency win evaporates.
When any one of these is absent, a single co-located stack running continuous batching plus chunked prefill is usually the better answer. Chunked prefill in particular (Sarathi-Serve style) bounds the impact of long prompts on decode latency without the operational complexity of two pools.
The right reading: papers like DistServe and Splitwise are valuable inputs for designing a serving stack, but the recommendation to deploy disaggregated is workload-specific and traffic-specific, not universal.
Quick reference
TL;DR
Disaggregating prefill and decode pays off only when prompts are long, traffic is high enough to saturate both pools, and the KV-transfer link between them is fast. Miss any one and co-located continuous batching wins.
Imagine a kitchen with one chef who chops vegetables and cooks dishes on the same stove. When orders pour in for elaborate dishes that need lots of prep, the chopping starts blocking the cooking and meals come out late. So you split the kitchen into a prep station and a cooking station with a conveyor belt between them. That helps only if three things are true: the prep work is genuinely heavy, there are enough orders to keep both stations busy, and the conveyor moves prepped ingredients fast enough that the cook is not standing around waiting. If any one of those is off, the original single-station setup with smart scheduling beats the split kitchen on cost and capacity. Disaggregating prefill and decode is the same tradeoff at the GPU level: useful at scale, wasteful below it.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short-prompt, high-decode workloads like chat with short user turns Prefill is already a thin slice of step time, so isolating it adds operational complexity without measurable latency benefit.
Modest QPS where neither pool can stay saturated One pool sits idle while the other queues, lowering effective utilization below what a flexible co-located stack would deliver.
Deployments without NVLink-Switch or fast RDMA between pools Shipping multi-GB KV caches per request over slow fabric eats the latency budget the architecture was meant to save.
Quick scan · 2–4 min
Everything important, quickly.
5 min: phase imbalance + three gating conditions + KV transfer cost + chunked prefill alternative + when disaggregation actually wins.
Real-world examples
Real products, models, and research that use this idea.
1DistServe (OSDI 2024) introduced the disaggregated prefill / decode design and quantified the gains on long-context workloads.
2Microsoft's Splitwise paper (ISCA 2024) showed similar gains and proposed phase-specific GPU tiers, prefill on H100s and decode on cheaper memory-rich SKUs.
3vLLM v1 added a disaggregated mode in 2025 and Anyscale's serving docs explicitly call out the three preconditions before recommending it.
4Anthropic and DeepSeek both deploy phase-aware scheduling in their serving stacks, with internal reports indicating disaggregation is reserved for long-context and high-QPS routes.
5Sarathi-Serve (Microsoft Research 2024) demonstrated that chunked prefill on a single pool captures most of the latency benefit at far lower operational cost, which is why vLLM v1 ships it as the default.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does chunked prefill (Sarathi-Serve style) capture much of the same benefit without disaggregation?
Sign in or go Pro to see the approach for this follow-up.
QHow would you size a disaggregated deployment given a target QPS and prompt length distribution?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes when one of the phases is offloaded to a cheaper GPU tier?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating disaggregation as a default architecture for serving large models. It is a scale-driven optimization with three preconditions; below those thresholds a co-located continuous-batching stack with chunked prefill is simpler and cheaper.