Describe RadixAttention as implemented in SGLang. Be specific about the data structure used, how prefix matches are detected across requests, and why this matters for agent and few-shot workloads.
RadixAttention indexes cached KV blocks in a persistent radix tree keyed by token prefix, so a new request automatically reuses the longest matching prefix and prefills only its divergent suffix.
Imagine a library where every book that starts with the same chapter shares one physical copy of that chapter, branching into separate copies only when the stories diverge. A radix tree is that shared shelf for token sequences. When a new reader arrives with a story, a clerk walks the shelf word by word, follows the path that matches, and hands over every shared chapter already on hand. The reader only has to write the new part their story adds. Because so many readers begin with the same long preface, the system rarely rewrites that preface twice. The walk takes about as long as the shared part, not as long as the whole library, so finding the match is almost free.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
RadixAttention is SGLang's answer to a simple but expensive observation: in real serving workloads, requests overlap heavily at the front. Agent loops resend the same system prompt and tool scaffolding every step. Few-shot pipelines prepend one long block of exemplars to every query. Tree-of-thought and branched sampling fork many continuations from a single shared context. In all of these, a large fraction of the prompt is identical across requests, and recomputing that identical chunk into the same keys and values is pure waste.
The naive KV cache is per request: each request gets its own keys and values and throws them away when it finishes. Prefix sharing improves on that by letting requests with a common opening reuse the same blocks. The hard part is detecting the overlap cheaply, across many concurrent and recently finished requests, without scanning the whole cache. RadixAttention solves detection by making the cache itself a prefix index.
This deep dive covers the radix tree structure, how matches are found and refcounted, how eviction stays correct, how it differs from vLLM, why cache-aware scheduling multiplies the gain, and which workloads turn this from a nice-to-have into a major throughput lever.
The radix tree as the cache index
A radix tree is a compressed prefix tree, also called a Patricia trie. Each edge carries a span of tokens, and each path from the root spells out a token sequence. RadixAttention keys this tree by the request's token sequence and attaches physical KV blocks to the nodes along each path. The compression matters: rather than one node per token, a long run of tokens with no branch point collapses into a single edge, so the tree stays shallow and the walk stays cheap even for very long prompts.
The consequence is that any two requests sharing an opening sequence share the same path, and therefore the same physical keys and values, up to the point they diverge. At the divergence point the path splits, and each branch owns the blocks for its own suffix. A third request whose prompt matches even deeper simply walks further down the existing path before branching, so the tree captures the longest common prefix automatically rather than only a fixed block boundary.
This is the key shift in framing. Prefix sharing is not a special case checked opportunistically. It is the default geometry of the cache. The tree is the single source of truth for what KV exists and what is shared, so reuse falls out of the structure rather than being bolted on after the fact. Because tokens are matched exactly, sharing is correctness-preserving: two prompts only merge when their tokens are identical, so the keys and values they would have computed are identical too.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- SGLang ships RadixAttention as its default KV cache manager, serving Llama 4 and Qwen 3 with automatic cross-request prefix sharing.
- Agent frameworks running long fixed system prompts and tool scaffolding on SGLang see large hit-rate gains because every loop step reuses the same prefix.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is reference counting necessary on radix tree nodes rather than plain LRU alone?
A live request still attending over a shared prefix must not have those blocks evicted. Reference counts mark nodes as in use, so LRU only reclaims leaves with zero references. Think about correctness under concurrent branching requests.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying RadixAttention is just vLLM prefix sharing with a new name. The defining difference is the persistent automatic cross-request radix tree versus per request hash lookups.
60 second bullets to scan on the way to the call.
The radix tree data structure and how token sequences map to paths
Why a tree walk is proportional to prefix length, not cache size
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.