Recognize vLLM and the optimization that made it famous

Attempt it

vLLM is an open-source LLM serving engine originally built at UC Berkeley. The optimization it is best known for is PagedAttention, a block-based KV-cache allocator inspired by OS virtual memory paging. Naive contiguous allocators waste tens of percent of HBM to internal fragmentation when sequence lengths vary; PagedAttention eliminates that waste, which lets vLLM pack many more concurrent sequences onto the same GPU and was the headline reason its throughput leapt past earlier servers in 2023. Continuous batching and prefix caching round out the picture, but PagedAttention is the signature feature.

Your answer

**vLLM** is the open-source LLM serving engine originally built at UC Berkeley in 2023. The name's 'v' nods to the Berkeley systems lineage; the rest of the acronym just says LLM. It is one of the three serving stacks every modern inference engineer is expected to know (the other two are TGI and TensorRT-LLM). What made vLLM famous is a single optimization called **PagedAttention**. Before vLLM, serving stacks allocated the KV cache as one contiguous slab per sequence, sized for the model's maximum context length. Real traffic has variable sequence lengths, so most of those reservations sat empty. Fragmentation was costing tens of percent of HBM. PagedAttention borrowed the operating-system virtual-memory paging idea. It split the KV cache into fixed-size blocks (typically 16 tokens), gave each sequence a block table that pointed to its blocks, and admitted new sequences from a shared free pool. HBM packed densely, batches stayed full, and reported throughput jumped roughly an order of magnitude on real workloads. vLLM also ships continuous batching, prefix caching, and FP8 quantization, but PagedAttention is the signature feature and the answer to 'why is vLLM fast?'

**vLLM** is the open-source LLM serving engine that came out of UC Berkeley in 2023. It became the default open-source choice for production LLM serving almost immediately on release. The single optimization vLLM is associated with is **PagedAttention**. The framing in the launch paper was direct: existing serving stacks were wasting 60 to 80 percent of HBM to KV-cache fragmentation. Their contiguous-allocation strategy forced every sequence to reserve max-context bytes up front, even if the sequence was short. PagedAttention fixed this by carving the KV cache into fixed-size blocks (typically 16 tokens), giving each sequence a block table mapping logical positions to physical block indices, and allocating blocks on demand from a global free pool. The mechanic is direct OS virtual-memory paging applied to attention's KV cache. The attention math is unchanged; only the addressing changes. The downstream effect is enormous: HBM packs densely, many more concurrent sequences fit per GPU, continuous batching becomes practical because new requests can be admitted mid-flight, and prefix caching becomes natural through reference-counted shared blocks. Reported throughput on benchmark workloads leaped roughly an order of magnitude versus earlier servers. PagedAttention also became influential beyond vLLM: TGI, TensorRT-LLM (as 'in-flight batching'), and SGLang (as RadixAttention) all implement variations of the same idea. Continuous batching, prefix caching, and FP8 quantization are also vLLM features, but they were enabled by PagedAttention and remain downstream of it.

Quick reference

TL;DR

vLLM is the open-source LLM serving engine from UC Berkeley, best known for PagedAttention, a block-based KV-cache allocator that eliminated fragmentation and lifted throughput an order of magnitude.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a library where every reader gets a private shelf sized for the longest book in the collection, even if they only borrow a slim novel. The library fills up fast and many readers wait outside. vLLM rearranged the shelves into uniform cubbies and gave each reader only the cubbies they actually use, picked from anywhere in the building. A central clipboard remembers which cubbies belong to whom. Suddenly the library fits many more readers. That clipboard trick is PagedAttention, and it is why vLLM became the default open-source serving engine almost overnight when it launched in 2023.

Key concepts

vLLM PagedAttention Continuous batching KV cache Serving frameworks

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: Berkeley origin + PagedAttention as signature feature + OS-paging analogy + fragmentation it eliminates + enabled continuous batching and prefix caching + roughly 10x throughput jump over earlier servers.

Real-world examples

Real products, models, and research that use this idea.

vLLM is the default serving engine for Llama 3.1, Mistral Large 3, DeepSeek V4, and Qwen 3.5 in most open-source production deployments.
Anyscale (Ray Serve) ships vLLM as its primary LLM inference backend at scale.
Many self-hosted Claude-like alternatives running Llama 4 Maverick or Scout in 2026 default to vLLM on H100 fleets.
vLLM benchmarks reported roughly 10x to 20x throughput improvements over earlier servers when the project launched.
Hugging Face TGI adopted block-based KV allocation directly inspired by vLLM's PagedAttention design.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does vLLM implement prefix caching on top of PagedAttention?

Sign in or go Pro to see the approach for this follow-up.

QWhat is vLLM's continuous batching scheduler doing each step?

Sign in or go Pro to see the approach for this follow-up.

QHow does vLLM compare against TensorRT-LLM on the same hardware?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying vLLM is just continuous batching. Continuous batching existed before; PagedAttention is what unlocked it at production scale.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.