Which workload most benefits from RadixAttention specifically vs simpler per request prefix caching?
RadixAttention wins where many requests share a branching prefix (agent trees, few-shot fan-out) because a radix tree reuses KV across all branches, not just one linear prefix per request.
Imagine a tour guide who memorizes the start of a story so every visitor hears the opening for free. A simple cache helps when one visitor asks the same thing twice. But agents are like a choose-your-own-adventure book: everyone reads the same first chapters, then each picks a different branch, and some branches loop back together. RadixAttention is a smart librarian who maps the whole book as a tree. Every reader who shares chapters reuses the same bookmark, no matter where they branch. So the shared opening is computed once and handed to many branching readers at once. The bigger and bushier the tree of shared beginnings, the more work the librarian saves for everyone.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
RadixAttention is SGLang's mechanism for automatic, cross-request reuse of the KV cache, built on a radix tree over token sequences. The interview question is deliberately narrow: it does not ask whether prefix caching helps in general, it asks which workload benefits from RadixAttention specifically, as opposed to the simpler per-request or hash-based prefix caching that vLLM ships.
That framing is the whole test. Many candidates know that reusing a shared prefix saves prefill compute. Fewer can articulate why a radix tree is the right data structure, and which sharing pattern actually exercises it. The key insight is that RadixAttention's advantage is not linear prefix reuse, which a hash cache already does well. Its advantage is branching prefix reuse, where one trunk fans out into many overlapping continuations that may reconverge.
The distinction matters because the cost of a cache is not just hit rate, it is also the bookkeeping the index imposes. A simpler hash cache is cheaper to maintain but blind to partial overlap. A radix tree is more expensive to maintain but expressive enough to share KV across an entire family of branching requests. The right answer therefore turns on a structural property of the workload: is the sharing pattern a single line, or a tree?
This deep dive walks through what RadixAttention is, how the tree works, how it differs from block-hash caching, why the branching agent loop is the correct answer, and why each distractor fails. By the end you should be able to defend the choice on structure, not just name-drop the feature.
What RadixAttention actually is
RadixAttention does not change the attention computation. The kernel still computes scaled dot-product attention between the new token's query and the cached keys and values. What it changes is how the KV cache is indexed and shared across requests. This is worth stressing in an interview, because candidates often imagine a clever new math trick when the innovation is really a memory-management policy.
A radix tree is a compressed prefix tree where each edge holds a run of tokens rather than a single token. Each node corresponds to a contiguous span of cached KV. The path from the root to any node spells out a token sequence whose K and V are already computed and resident in HBM. Because edges hold variable-length runs, the tree stays compact even when prefixes are thousands of tokens long.
When a new request arrives, the runtime walks the tree from the root, following edges whose tokens match the request. It reuses the KV along the matched path for free, skipping that much prefill. Where the request diverges, the runtime splits the edge and adds a new leaf for the unique tail. The shared trunk is computed exactly once regardless of how many requests traverse it.
The consequence is that prefill work for a batch of related requests collapses toward the size of the union of their token sequences, rather than the sum. The more the requests overlap, the larger the gap between those two quantities, and the larger the win.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | vLLM hash prefix cache | SGLang RadixAttention |
|---|---|---|
| Index structure | Hash of fixed-size token blocks | Radix tree over token sequences |
| Best case | One stable repeated linear prefix | Many branches sharing a common trunk |
| Branching support | Limited; per-block exact match | Native; fork edges where requests diverge |
| Reconvergence | Not represented | Captured as shared nodes |
| Ideal workload | Shared system prompt across users | Agent loops, tree search, few-shot fan-out |
Real products, models, and research that use this idea.
- SGLang serves Llama 4 and Qwen 3 agent workloads with RadixAttention reusing shared system-prompt and reasoning prefixes across branching tool calls.
- Tree-of-thought and self-consistency pipelines fan out many sampled continuations from one prompt, where the radix tree shares the common trunk's KV.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does a radix tree match a new request to the longest shared prefix?
Walk from the root, following edges whose tokens match the incoming sequence. Reuse KV along the matched path. Where the next token diverges, split the edge and create a new leaf for the unique tail. Match length determines prefill saved.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming any prefix caching equals RadixAttention. Single user reuse is handled by a plain hash cache. The radix tree pays off only when many branching requests share overlapping prefixes.
60 second bullets to scan on the way to the call.
What a radix tree indexes and how a request walks it
Why branching prefixes favor a tree over a hash cache
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.