Name the architectural shifts vLLM v1 made relative to v0

Attempt it

vLLM v1 (2024-2025) is a significant rewrite of the v0 engine, not just a feature release. Name the three architectural shifts that define v1, and explain for each what concrete problem in v0 it solves.

Free · 2 AI evals / day

**vLLM v1** is a ground-up rewrite of the v0 engine that shipped through 2024 and 2025. The rewrite was motivated by three concrete production pains. **1. Unified scheduler with chunked prefill plus continuous decode.** v0 ran prefill and decode in separate phases. When a long prompt arrived, every in-flight decode paused while the long prefill ran end to end. P99 TPOT spiked exactly when a long prompt landed. v1 slices long prefills into chunks and interleaves them with decode steps in one scheduling tick. The long prompt takes a little longer to first token, but no other request slows down. **2. Multi-process architecture.** v0 kept the API server, scheduler, and model runner in one Python process. The Python GIL serialised request handling against CUDA dispatch, and CUDA graph capture fought with scheduling for the GPU stream. v1 moves the model runner into a separate process, talking to the scheduler over shared memory or ZMQ. The scheduler can keep accepting requests while CUDA graphs replay cleanly. **3. Block manager with first-class prefix caching.** v0's prefix cache was bolted on top of PagedAttention, with awkward sharing and fragmentation under bursty traffic. v1's block manager treats prefix sharing as a primitive, with content-hashed and reference-counted KV blocks. A system prompt shared by 1000 sessions occupies one set of blocks, not 1000 copies, freeing HBM for bigger batches. Supporting shifts include torch.compile integration, a cleaner CUDA/ROCm/TPU backend abstraction, and lower per-request overhead at high concurrency.

Three shifts define v1. **Unified scheduler with chunked prefill plus continuous decode.** v0 split prefill and decode into separate phases. A 16k-token prefill arriving during active serving blocked every in-flight decode until it finished, sending P99 TPOT through the roof. v1 slices long prefills into configurable chunks (typically 256 to 512 tokens) and schedules those chunks alongside decodes for other requests in one tick. TTFT for the long prompt goes up modestly; everyone else's TPOT stays flat. This matches the SARATHI / chunked-prefill design and is now the convergence design across vLLM, SGLang, and TensorRT-LLM. **Multi-process architecture (API plus model-runner split).** v0 ran everything in one Python process. The GIL serialised the scheduler against CUDA dispatch, and CUDA graph capture had to share the GPU stream with scheduling work. v1 puts the model runner in a dedicated process, communicating with the scheduler over shared memory or ZMQ. The scheduler can accept and admit requests while the runner replays CUDA graphs without contention. Graph capture becomes the default mode rather than an opt-in flag, which is worth several percent throughput on its own. **Block manager with first-class cross-request prefix caching.** v0's prefix cache layered on top of PagedAttention as an afterthought, with awkward sharing semantics and KV-block fragmentation under bursty workloads. v1 redesigns the block manager so prefix sharing is a primitive: content-hashed block keys, reference-counted blocks, automatic deduplication across requests. The practical effect at production scale is dramatic. A 30k-token system prompt shared by 1000 active sessions occupies one set of KV blocks instead of 1000, freeing tens of GB of HBM for larger effective batch sizes. Secondary shifts: torch.compile integration replaces hand-written CUDA for several layers, the backend abstraction unifies CUDA / ROCm / TPU paths, and request/sequence dataclasses are rewritten for

Quick reference

TL;DR

vLLM v1 unified prefill+decode in one scheduler (kills head-of-line blocking), split scheduler from model runner across processes (frees the GIL and CUDA graphs), and made cross-request prefix caching a first-class

Memory aid

Easy to grasp

Imagine a kitchen that used to cook one big pot at a time, with everyone waiting until it finished, on a single chef trying to take orders and cook at the same time, in a pantry where every customer got their own copy of the bread basket. The new kitchen interleaves a slice of the big pot with the small pans every minute so nobody waits long, hires a separate waiter so the chef can focus on cooking, and shares one bread basket across customers who ordered the same starter. Same kitchen, three structural changes, dramatically more orders served per hour. vLLM v1 is exactly that redesign: smarter scheduling, separated roles, and shared resources where it is safe to share.

Key concepts

vLLM Chunked prefill PagedAttention Prefix caching CUDA graphs

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: the three architectural shifts (unified scheduler, process split, block manager prefix cache) + each shift's v0 pain point + supporting shifts (torch.compile, backend abstraction, dataclass rewrite) + combined operational profile.

Real-world examples

Real products, models, and research that use this idea.

vLLM v1 chunked prefill matches the SARATHI design and the TRT-LLM 0.18 executor mode, the cross-engine convergence pattern in 2026.
Anthropic, OpenAI, and other API providers ship prompt caching products that surface the same engine-level primitive vLLM v1 made first-class.
DeepSeek and Mistral run vLLM v1 in production with multi-host tensor parallelism, leaning on the cleaner backend abstraction.
vLLM v1 ROCm path on AMD MI300X reached production parity with the CUDA path through 2025, a direct benefit of the backend rewrite.
SGLang's RadixAttention shares the same first-class prefix-caching design philosophy and demonstrates the convergence across major engines.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does chunked prefill interact with CUDA graph capture in v1?

QWhy is reference counting the right primitive for the block manager?

QWhat changes about multi-GPU and multi-host scaling in v1?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating v1 as a feature release rather than an architecture rewrite. The three shifts are structural: unified scheduling, process split, and block-manager prefix caching are not toggles, they are different shapes of the engine.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.