What problem do DistServe and Splitwise solve by separating prefill and decode onto different GPUs?

Attempt it

Describe the problem with co-locating prefill and decode on the same GPU pool, and how disaggregated inference (DistServe, Splitwise) fixes it. What is the cost?

Free · 2 AI evals / day

Quick reference

TL;DR

Disaggregation puts prefill and decode on separate GPU pools so a long compute-bound prefill can no longer stall every concurrent user's decode, at the cost of one KV-transfer hop per request.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a restaurant where one chef both cooks big banquet orders and plates the steady stream of small dishes. A giant banquet order ties up the kitchen, and every small dish waits behind it. Customers expecting quick service get annoyed. The fix: hire two teams. One team only handles the big bursty prep work, the other only plates the steady stream of small dishes. Now a huge order cannot freeze the quick service. The only new chore is carrying the prepped ingredients from the first team to the second. As long as that handoff is fast, every customer gets smooth, predictable service. That handoff is exactly the KV cache moving from the prefill machines to the decode machines.

Key concepts

Prefill/decode disaggregation KV cache Time per output token Continuous batching

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: prefill vs decode bottlenecks + the interference pathology on a shared pool + the two-pool fix + KV transfer over NVLink and RDMA + when it pays off versus the hop cost.

Real-world examples

Real products, models, and research that use this idea.

DistServe (OSDI 2024) disaggregates prefill and decode, reporting large goodput gains under per-phase latency SLOs.
Microsoft's Splitwise splits the phases across pools and shows you can use cheaper, older GPUs for the decode tier.
NVIDIA Dynamo (2025) ships disaggregated serving as a first-class deployment mode for Llama 4 and DeepSeek V4 class models.
vLLM and SGLang both added prefill/decode disaggregation with KV transfer over NVLink and RDMA in their 2025 releases.
Mooncake (Moonshot AI, Kimi) runs a disaggregated KV-cache-centric architecture with a pooled KV store across prefill and decode nodes.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you size the prefill pool versus the decode pool, and what breaks when traffic shifts?

Sign in or go Pro to see the approach for this follow-up.

QWhen does the KV-transfer hop dominate and kill the benefit?

Sign in or go Pro to see the approach for this follow-up.

QHow does disaggregation interact with prefix caching and continuous batching?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming disaggregation is always faster. It only pays off under high traffic with mixed prompt lengths and tight latency targets; at low load the extra KV hop just adds latency.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.