Which workload trace shows the strongest case for disaggregated prefill/decode serving?
Disaggregation pays off only when prefills are both large and frequent enough to stall decodes: long prompts at high concurrency, the RAG and agent regime.
Picture one chef doing two very different jobs. Sometimes a huge catering order arrives and takes all his attention for several minutes. Meanwhile, dozens of customers want one bite-sized appetizer every few seconds, served smoothly. If the chef stops to cook the catering order, every appetizer customer waits and gets annoyed. The fix is two kitchens: one chef handles big catering bursts, the other keeps the steady trickle of appetizers flowing. This only helps if big orders are both large and arrive often. If catering orders are tiny, or almost never arrive, splitting the kitchens just adds overhead and a handoff cost between them.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Disaggregated serving is one of the highest-leverage architectural moves in 2026 LLM inference, and it is a favorite senior-level interview probe because it forces you to reason about two phases with opposite bottlenecks at once. The question is not whether disaggregation is good, but for which workload it actually pays. That is a tradeoff question, and the wrong answers are all plausible until you apply the test.
The core fact: prefill and decode are different computations sharing the same model. Prefill ingests the whole prompt in one parallel pass and is compute-bound. Decode emits one token at a time, streaming the KV cache from HBM, and is memory-bandwidth bound and latency-sensitive. On a single GPU they compete, and a long prefill burst stalls every concurrent decode.
This deep dive works through why the two phases interfere, what disaggregation does about it, the precise two-part condition under which it earns its complexity, why three of the four traces fail that condition, the KV cache transfer cost you must always price in, and the goodput framing that explains the whole design. The recurring theme is that disaggregation is a targeted fix for one specific pathology, not a free upgrade, so the interview reward goes to whoever names the regime and the cost rather than reciting the mechanism. By the end you should be able to look at any production trace and predict whether splitting the pools will help, and articulate the latency contract that makes the answer concrete.
The two phases and why they collide
Every generation request runs prefill then decode. Prefill processes all input tokens in parallel, fills the KV cache, and produces the first output token. It is a dense burst of matmuls that saturates the GPU's compute units; attention cost grows with the square of prompt length, so an 8k-to-32k prompt is genuinely heavy.
Decode is the autoregressive tail. Each step computes one new token, reads the entire KV cache from HBM, and is bound by memory bandwidth rather than compute. What matters to users here is inter-token latency, the smoothness of the stream.
When both phases land on the same GPU, the heavy prefill burst monopolizes compute for hundreds of milliseconds. Every decode in flight at that moment stalls, spiking the inter-token latency of unrelated users. The damage is a tail-latency problem, not an average-throughput one, which is exactly what production targets care about.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DistServe disaggregates prefill and decode onto separate GPU pools and tunes each phase to its own latency target, the canonical academic system for this design.
- Microsoft's Splitwise splits the two phases across machine pools in production Azure LLM serving and reports higher throughput under fixed latency targets.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is chunked prefill sometimes a sufficient alternative to full disaggregation?
Chunked prefill slices a long prompt into small windows that interleave with decode tokens on the same GPU. This bounds the per-step stall without paying a KV transfer. It can satisfy latency targets when prefill pressure is moderate, but it still consumes shared compute, so it loses to disaggregation under sustained heavy prefill.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking the long-completion creative-writing trace. That workload is almost all decode with trivial prefill, so there is nothing to disaggregate; the collision that disaggregation removes never happens.
60 second bullets to scan on the way to the call.
Why prefill and decode have opposite bottlenecks
The two-part interference test for when disaggregation pays off
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.