How does PagedAttention enable prefix sharing across requests and what data structure makes it cheap?

Attempt it

Explain how PagedAttention's block-table design lets multiple in-flight requests share KV cache for a common prompt prefix without copying it. What data structure makes detection cheap, and what happens when one request needs to diverge from the shared prefix?

Free · 2 AI evals / day

Quick reference

TL;DR

Block tables map each request's KV positions to fixed-size physical blocks, so requests with a shared prefix point at the same blocks; a hash index detects the match and copy-on-write handles divergence.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a library where every reader needs the same opening chapters of a textbook before branching into their own topic. Instead of photocopying those chapters for each reader, the librarian hands everyone a card that points to the one shared copy on the shelf. The library keeps a tally of how many readers point at each shelf slot. When a reader finally wants to scribble their own notes onto a page, the librarian photocopies just that one page, gives it to them, and updates their card to point at the copy. Everyone else keeps sharing the original. A quick lookup table tells the librarian instantly whether an arriving reader's opening chapters already sit on a shelf, so nobody waits while shared pages are found and reused.

Key concepts

PagedAttention Prefix caching Copy-on-write KV cache

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: block table indirection + pointer sharing of prefix blocks + reference counting + copy-on-write divergence + parent-chained block hashing + RadixAttention contrast + eviction and cache-key correctness.

Real-world examples

Real products, models, and research that use this idea.

vLLM ships automatic prefix caching that hashes KV blocks and reuses them across requests; it is the default serving engine for most open-source Llama 4 and Qwen 3 deployments in 2026.
SGLang's RadixAttention stores cached prefixes in a radix tree, giving longest-prefix match and LRU eviction for agent and few-shot workloads.
Anthropic and OpenAI expose prompt caching on their APIs, billing cached prefix tokens at a fraction of normal input cost for repeated system prompts.
TensorRT-LLM implements block reuse and KV cache reuse on NVIDIA H100 and B200 for shared system-prompt serving.
RAG and agent stacks with a multi-thousand-token shared instruction block see prefill for that span computed once rather than once per request under vLLM prefix caching.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy must the block hash chain the parent block's hash rather than hashing tokens alone?

Sign in or go Pro to see the approach for this follow-up.

QContrast vLLM's hash-table prefix cache with SGLang's RadixAttention tree.

Sign in or go Pro to see the approach for this follow-up.

QWhat besides token IDs must enter the cache key for correctness?

Sign in or go Pro to see the approach for this follow-up.

QHow does eviction interact with reference counting so shared prefixes stay safe?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying the shared prefix is copied into each request's cache. It is referenced by pointer, never duplicated; only a divergent block is ever copied, and only that block.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.