Name the architectural shifts vLLM v1 made relative to v0

Short answer·Medium·4.0 · 0·~3 min·Asked atAutodesk Groq Meesho·Relevant atVllm

Attempt it

vLLM v1 (2024-2025) is a significant rewrite of the v0 engine, not just a feature release. Name the three architectural shifts that define v1, and explain for each what concrete problem in v0 it solves.

Free · 2 AI evals / day

**vLLM v1** is a ground-up rewrite of the v0 engine that shipped through 2024 and 2025. The rewrite was motivated by three concrete production pains. **1. Unified scheduler with chunked prefill plus continuous decode.** v0 ran prefill and decode in separate phases. When a long prompt arrived, every in-flight decode paused while the long prefill ran end to end. P99 TPOT spiked exactly when a long prompt landed. v1 slices long prefills into chunks and interleaves them with decode steps in one scheduling tick. The long prompt takes a little longer to first token, but no other request slows down. **2. Multi-process architecture.** v0 kept the API server, scheduler, and model runner in one Python process. The Python GIL serialised request handling against CUDA dispatch, and CUDA graph capture fought with scheduling for the GPU stream. v1 moves the model runner into a separate process, talking to the scheduler over shared memory or ZMQ. The scheduler can keep accepting requests while CUDA graphs replay cleanly. **3. Block manager with first-class prefix caching.** v0's prefix cache was bolted on top of PagedAttention, with awkward sharing and fragmentation under bursty traffic. v1's block manager treats prefix sharing as a primitive, with content-hashed and reference-counted KV blocks. A system prompt shared by 1000 sessions occupies one set of blocks, not 1000 copies, freeing HBM for bigger batches. Supporting shifts include torch.compile integration, a cleaner CUDA/ROCm/TPU backend abstraction, and lower per-request overhead at high concurrency.

Three shifts define v1. **Unified scheduler with chunked prefill plus continuous decode.** v0 split prefill and decode into separate phases. A 16k-token prefill arriving during active serving blocked every in-flight decode until it finished, sending P99 TPOT through the roof. v1 slices long prefills into configurable chunks (typically 256 to 512 tokens) and schedules those chunks alongside decodes for other requests in one tick. TTFT for the long prompt goes up modestly; everyone else's TPOT stays flat. This matches the SARATHI / chunked-prefill design and is now the convergence design across vLLM, SGLang, and TensorRT-LLM. **Multi-process architecture (API plus model-runner split).** v0 ran everything in one Python process. The GIL serialised the scheduler against CUDA dispatch, and CUDA graph capture had to share the GPU stream with scheduling work. v1 puts the model runner in a dedicated process, communicating with the scheduler over shared memory or ZMQ. The scheduler can accept and admit requests while the runner replays CUDA graphs without contention. Graph capture becomes the default mode rather than an opt-in flag, which is worth several percent throughput on its own. **Block manager with first-class cross-request prefix caching.** v0's prefix cache layered on top of PagedAttention as an afterthought, with awkward sharing semantics and KV-block fragmentation under bursty workloads. v1 redesigns the block manager so prefix sharing is a primitive: content-hashed block keys, reference-counted blocks, automatic deduplication across requests. The practical effect at production scale is dramatic. A 30k-token system prompt shared by 1000 active sessions occupies one set of KV blocks instead of 1000, freeing tens of GB of HBM for larger effective batch sizes. Secondary shifts: torch.compile integration replaces hand-written CUDA for several layers, the backend abstraction unifies CUDA / ROCm / TPU paths, and request/sequence dataclasses are rewritten for

TL;DR

vLLM v1 unified prefill+decode in one scheduler (kills head-of-line blocking), split scheduler from model runner across processes (frees the GIL and CUDA graphs), and made cross-request prefix caching a first-class

Memory aid

Easy to grasp

Imagine a kitchen that used to cook one big pot at a time, with everyone waiting until it finished, on a single chef trying to take orders and cook at the same time, in a pantry where every customer got their own copy of the bread basket. The new kitchen interleaves a slice of the big pot with the small pans every minute so nobody waits long, hires a separate waiter so the chef can focus on cooking, and shares one bread basket across customers who ordered the same starter. Same kitchen, three structural changes, dramatically more orders served per hour. vLLM v1 is exactly that redesign: smarter scheduling, separated roles, and shared resources where it is safe to share.

Key concepts

vLLM Chunked prefill PagedAttention Prefix caching CUDA graphs

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

vLLM v1 is one of the most consequential serving-engine rewrites of the 2024 to 2025 window. The v0 engine, which shipped PagedAttention and made continuous batching mainstream, was the dominant open-source serving stack from 2023 onward. By 2024 the production pain in v0 had accumulated to the point where incremental fixes were no longer enough, and the maintainers committed to a structural redesign. v1 shipped in stages through 2024 and 2025 and is now the default in 2026.

Understanding what v1 changed requires looking past the feature list to the three structural shifts that motivated the rewrite. Each shift exists to eliminate a specific class of production failure that v0 could not address without touching the engine's core architecture. Each shift also has a name and a clear v0 antecedent, which makes the question scoring straightforward: a complete answer names all three shifts and explains the v0 pain each one resolves.

This deep dive walks through each shift in detail, including the v0 failure mode it eliminates, the mechanics of the v1 replacement, and the operational metrics that move when you migrate. It also covers the secondary shifts (torch.compile, backend abstraction, dataclass rewrite) that round out the rewrite, and closes with the combined operational profile that explains why production deployments migrated despite the migration cost. The goal by the end is to be able to read the vLLM v1 release notes correctly and explain to a colleague exactly what changed at the architectural level.

Shift 1: unified scheduler with chunked prefill plus continuous decode

v0's pain. The scheduler operated in two phases per tick: a prefill phase that processed newly admitted prompts through their full input length, and a decode phase that advanced each active sequence by one token. When a 16k-token prefill landed during a tick, the entire prefill ran in that phase, and every in-flight decode froze until the prefill finished. On a 70B model that freeze could be 200 to 400 ms, blowing P99 TPOT for every concurrent user.

The pain compounded under bursty traffic. A few long prompts arriving close together stacked their prefill phases, and P99 TPOT could climb into the seconds during traffic spikes. Operators worked around this by capping prompt length, splitting traffic onto separate replicas by length class, or scaling out beyond the actual capacity needs. None of these workarounds was clean; the underlying scheduler design was wrong.

v1's redesign. The scheduler runs a single tick per scheduling step, not two phases. Long prefills are sliced into chunks of a configurable size (typical defaults 256 to 512 tokens), and the scheduler interleaves prefill chunks with decode steps in the same tick. The chunk size is the knob: smaller chunks reduce decode disruption but increase prefill overhead. Production tuning usually lands near 256 to 512 tokens, balancing the two.

The operational effect is dramatic. P99 TPOT under mixed prefill/decode load drops from spiky to flat. TTFT for long prompts goes up modestly (the prefill takes a few more ticks because each tick processes only one chunk), but the trade is widely accepted in production because tail TPOT matters more for user experience.

This design is now the convergence pattern. SARATHI proposed it academically in 2023, vLLM v1 productionised it, SGLang and TRT-LLM 0.18 followed. The phase-split design of v0 is gone industry-wide.

Shift 2: multi-process architecture

Shift 3: block manager with first-class prefix caching

Secondary shifts and combined operational profile

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM v1 chunked prefill matches the SARATHI design and the TRT-LLM 0.18 executor mode, the cross-engine convergence pattern in 2026.
Anthropic, OpenAI, and other API providers ship prompt caching products that surface the same engine-level primitive vLLM v1 made first-class.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does chunked prefill interact with CUDA graph capture in v1?

CUDA graphs require fixed shapes per replay. Chunked prefill uses a small set of fixed chunk shapes (e.g. 256 and 512 tokens), so the engine can capture graphs for each shape and pick the right one at scheduling time. The multi-process split is what makes this practical: the runner captures graphs once and replays them without scheduling-side interference.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating v1 as a feature release rather than an architecture rewrite. The three shifts are structural: unified scheduling, process split, and block-manager prefix caching are not toggles, they are different shapes of the engine.

60 second bullets to scan on the way to the call.

What head of line blocking looked like in v0 prefill/decode phases
How chunked prefill interleaves with decode in v1's unified scheduler

Primary sources. Browse if you want the original framing.