Describe RadixAttention as implemented in SGLang. Be specific about the data structure used, how prefix matches are detected across requests, and why this matters for agent and few-shot workloads.
Free · 2 AI evals / day
**RadixAttention** is SGLang's way of reusing the KV cache across many requests automatically. The KV cache holds the keys and values computed for tokens you have already processed. If two requests start with the same tokens, those tokens produce the exact same KV, so recomputing them is wasted work.
The trick is the data structure. SGLang keeps a radix tree, which is a compressed prefix tree, where each path from the root spells out a token sequence. Requests that share a prefix share the same path and therefore the same physical KV blocks. Where they differ, the path branches.
When a new request comes in, SGLang walks its prompt down the tree token by token. Every block along the matching path is reused, and only the part that diverges gets freshly prefilled. The walk costs time proportional to the prefix length, not the size of the whole cache.
This matters most for agent loops, few-shot prompts, and branching tool calls, where many requests share a long common chunk. vLLM also reuses matching prefixes, but RadixAttention makes the sharing automatic across all live and recently finished requests, which lifts the cache hit rate.
**RadixAttention** stores KV cache blocks in a radix tree keyed by token sequence. Each path from the root represents a prefix. Shared prefixes share one path and the same physical blocks. Where prompts diverge, the path branches.
When a request arrives, SGLang walks its tokens down the tree. Every matching node bumps a reference count and its KV blocks are reused. Only the divergent suffix is prefilled. The match costs time proportional to the prefix length, not the cache size, so detection is effectively free. Eviction is LRU at the leaf level, so cold branches are reclaimed first while hot shared prefixes stay resident.
The contrast with vLLM matters. vLLM also reuses blocks for matching prefixes, but historically the match was per request, often via block hashing. RadixAttention keeps one persistent tree spanning all in-flight and recently finished requests, so sharing is automatic and global.
SGLang also adds cache-aware scheduling. The scheduler routes requests that share a prefix to run together. That maximizes reuse before any branch is evicted. The big wins land on agent scaffolding, shared few-shot exemplars, and branching tool calls.
Quick reference
TL;DR
RadixAttention indexes cached KV blocks in a persistent radix tree keyed by token prefix, so a new request automatically reuses the longest matching prefix and prefills only its divergent suffix.
Imagine a library where every book that starts with the same chapter shares one physical copy of that chapter, branching into separate copies only when the stories diverge. A radix tree is that shared shelf for token sequences. When a new reader arrives with a story, a clerk walks the shelf word by word, follows the path that matches, and hands over every shared chapter already on hand. The reader only has to write the new part their story adds. Because so many readers begin with the same long preface, the system rarely rewrites that preface twice. The walk takes about as long as the shared part, not as long as the whole library, so finding the match is almost free.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Workloads where every request has a unique prompt with no shared prefix at all With no common prefix the tree never matches, so you pay tree bookkeeping for zero reuse.
Tiny prompts and short generations where prefill is already cheap The reuse saved is small, so tree maintenance overhead can exceed the prefill it avoids.
Severely memory-constrained deployments under constant eviction pressure If hot prefixes get evicted before reuse, the hit rate collapses and the tree stops paying off.
Pipelines that already pin a known shared prefix with explicit prompt caching Explicit prefix pinning gives deterministic reuse, so the automatic tree adds little beyond it there.
Quick scan · 2–4 min
Everything important, quickly.
3 min: radix tree keyed by token prefix + O(prefix) tree-walk match + reference counting and LRU eviction + vLLM hashing contrast + cache-aware scheduling + agent and few-shot wins.
Real-world examples
Real products, models, and research that use this idea.
1SGLang ships RadixAttention as its default KV cache manager, serving Llama 4 and Qwen 3 with automatic cross-request prefix sharing.
2Agent frameworks running long fixed system prompts and tool scaffolding on SGLang see large hit-rate gains because every loop step reuses the same prefix.
3Few-shot pipelines that prepend one long shared exemplar block reuse that entire block across thousands of requests via a single tree path.
4Tree-of-thought and branched sampling workloads fork from a shared context node, so RadixAttention shares the trunk and branches only the divergent suffixes.
5vLLM added automatic prefix caching with hash-based block matching, the closest comparison point interviewers expect you to contrast against.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is reference counting necessary on radix tree nodes rather than plain LRU alone?
Sign in or go Pro to see the approach for this follow-up.
QHow does cache-aware scheduling change the win compared to the tree alone?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the worst-case behavior of RadixAttention and how would you detect it in production?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying RadixAttention is just vLLM prefix sharing with a new name. The defining difference is the persistent automatic cross-request radix tree versus per request hash lookups.