Describe the problem with co-locating prefill and decode on the same GPU pool, and how disaggregated inference (DistServe, Splitwise) fixes it. What is the cost?
Disaggregation puts prefill and decode on separate GPU pools so a long compute-bound prefill can no longer stall every concurrent user's decode, at the cost of one KV-transfer hop per request.
Picture a restaurant where one chef both cooks big banquet orders and plates the steady stream of small dishes. A giant banquet order ties up the kitchen, and every small dish waits behind it. Customers expecting quick service get annoyed. The fix: hire two teams. One team only handles the big bursty prep work, the other only plates the steady stream of small dishes. Now a huge order cannot freeze the quick service. The only new chore is carrying the prepped ingredients from the first team to the second. As long as that handoff is fast, every customer gets smooth, predictable service. That handoff is exactly the KV cache moving from the prefill machines to the decode machines.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prefill/decode disaggregation is one of the highest-leverage architectural shifts in LLM serving since paged attention. It is also a favorite hard interview question because it forces the candidate to reason about two workloads with genuinely opposite hardware profiles sharing one accelerator, and to weigh a clean architectural fix against a real new cost.
The setup is simple once you see it. Every generation request runs a compute-bound prefill phase followed by a bandwidth-bound decode phase. Co-locating both phases on one GPU pool, the default in early serving stacks, means a long prefill and a stream of short decode steps fight for the same streaming multiprocessors. The loser is almost always decode, because it is short, steady, and latency-sensitive, while prefill is long and bursty.
This deep dive walks through why the two phases interfere, what disaggregation changes physically, how the KV cache moves between pools, what the move costs, and the precise conditions under which the trade actually pays off. By the end you should be able to argue both sides: when to disaggregate and when co-location is the right call.
Two phases, two bottlenecks
A request begins with prefill. The model ingests the entire prompt in one parallel forward pass, computing keys and values for every input token and emitting the first output token. This is dense matrix multiplication over a long sequence, so prefill saturates the GPU's compute units. It is compute-bound, and its latency grows with prompt length.
Then comes decode. The model generates one token at a time, each step reading the growing KV cache from high-bandwidth memory and producing a single new token. The arithmetic per step is tiny relative to the memory traffic, so decode is bandwidth-bound. Each step is short, roughly 10 to 30 ms, but the user feels the sum of all of them as the streaming speed of the response.
The two metrics that matter are TTFT, time to first token, which prefill governs, and TPOT, time per output token, which decode governs. The key fact for this question: prefill and decode stress different parts of the hardware, so a schedule that is good for one is rarely good for the other.
This asymmetry is not a minor implementation detail. It is the reason a single scheduling knob cannot satisfy both phases at once. Batch aggressively to feed the compute units during prefill and you lengthen the queue that decode steps must wait in. Prioritize decode latency and you leave the math units idle while prefill work piles up. The phases pull the same hardware in opposite directions, and that tension is exactly what disaggregation resolves by giving each phase its own machines.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DistServe (OSDI 2024) disaggregates prefill and decode, reporting large goodput gains under per-phase latency SLOs.
- Microsoft's Splitwise splits the phases across pools and shows you can use cheaper, older GPUs for the decode tier.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you size the prefill pool versus the decode pool, and what breaks when traffic shifts?
Drive the ratio from the workload's prompt to generation token mix and the per-phase service rates. Long-prompt RAG traffic needs more prefill capacity; chat traffic needs more decode. When the mix drifts, a static ratio strands GPUs, so production systems autoscale each pool independently or allow elastic borrowing.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming disaggregation is always faster. It only pays off under high traffic with mixed prompt lengths and tight latency targets; at low load the extra KV hop just adds latency.
60 second bullets to scan on the way to the call.
Why prefill and decode have opposite hardware bottlenecks
How a long prefill stalls concurrent decode on a shared pool
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.