Multi-turn chat has a specific cost structure. The first message in a conversation usually starts with a long system prompt (say, 2k tokens of instructions and persona), then a series of user and assistant turns. Every time a new turn arrives, the model has to process the **entire conversation history** as the input prompt, prefill its KV cache, and only then start generating the next response.
If you have many parallel users sharing the same system prompt (which is the production reality, every API user of a given assistant sees the same system prompt), the model is recomputing the **same KV cache** for that prefix again and again. This is wasted work.
The fix is **prefix caching**: store the KV cache for a prefix once, and reuse it whenever a new request shares that prefix. Two frameworks lead on this: vLLM v1 (general prefix caching) and **SGLang** (RadixAttention, which organizes prefixes in a radix tree to handle deep branching).
The other three options describe real but unrelated optimizations:
- **TensorRT-LLM**: CUDA graph capture, which speeds up fixed-shape execution by replaying recorded kernel launches. It is not about KV reuse across requests.
- **Triton Inference Server**: dynamic batching merges concurrent requests for batched execution. It is not about prefix sharing.
- **Hugging Face TGI**: tokenizers do not decide whether you recompute KV; that is a confusion of layers.
When the workload is 'lots of conversations sharing a system prompt and branching from there', SGLang's radix structure is the natural fit.
Multi-turn chat is the canonical workload for **prefix-aware KV reuse**, and the question turns on whether you can identify which framework has a first-class data structure for that.
**SGLang's RadixAttention** is a radix tree (a compressed trie) keyed by token IDs. Every prefix that has been seen has a node, and the KV blocks for that prefix live behind that node. When a new request comes in, the scheduler walks the tree to find the longest matching prefix and resumes from there. Two requests sharing a prefix of length P pay the prefill cost for P only once.
The **radix tree is the differentiator** from vLLM's prefix caching. vLLM v1 caches prefixes well in the common case (single prefix per system prompt), but RadixAttention shines when conversations **branch**: an agent forks the same system prompt into 10 sub-tasks, an A/B test sends two variants of the same prompt, a structured generator explores many continuations. The radix structure naturally handles sibling branches at any depth.
**TensorRT-LLM** uses CUDA graphs to amortize kernel launch overhead, which matters for low-latency single-request decode but is orthogonal to KV reuse across requests. **Triton Inference Server** (NVIDIA's general inference server) does dynamic batching, which merges concurrent requests for batched throughput, not for prefix dedup. **TGI** is solid for production deployment but its tokenizer is a tokenizer; nothing about a tokenizer decides whether you recompute KV.
So the framework choice maps directly to the workload's shape. For heavy shared-prefix multi-turn chat, RadixAttention is built for the case. vLLM v1 is the strong runner-up.
Quick reference
TL;DR
SGLang's RadixAttention indexes prompt prefixes in a radix tree so deeply branched chats reuse the same KV blocks; vLLM v1 has prefix caching too, but SGLang specifically targets branching agent workloads.
Imagine a thousand people at a library all reading the same first chapter of the same book, then branching to different second chapters. The slow way: every reader gets a fresh copy of chapter 1 photocopied just for them. The fast way: one shared copy of chapter 1 sits on a central table, and everybody reads from it before walking off to their own desk for their unique chapter 2. SGLang builds that shared table for you automatically. It looks at all the conversations coming in, finds the parts they share at the start (the system prompt, the user history), and only does the work once. The other frameworks listed either solve a different problem (faster kernel launches, faster HTTP routing, smarter tokenizers) or invent a feature that does not exist. The winning move is a data structure called a radix tree that organizes the shared prefixes.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Every request has a completely unique prompt (no shared system message) Prefix cache hit rate is zero. The framework choice should pivot to kernel performance and continuous batching instead.
Single-stream low-latency code completion with short prompts Kernel-launch overhead and decode bandwidth dominate; CUDA graph capture in TensorRT-LLM is the bigger lever.
Multi-model serving where you front several different models behind one endpoint Triton Inference Server as an orchestration layer (with SGLang or vLLM as the actual runtime) gives you model routing and version management that the runtimes themselves do not.
Quick scan · 2–4 min
Everything important, quickly.
1 min: name the workload's dominant cost (shared prefix prefill). 2 min: explain RadixAttention as a radix tree of token-id prefixes that lets sibling branches reuse KV. 1 min: contrast with vLLM v1 (hash table, single-prefix focused). 1 min: kill the wrong options by naming the wrong axis each addresses (CUDA graphs, dynamic batching, tokenizers).
Side-by-side
Framework
Prefix sharing strategy
Best for
Headline feature
SGLang
RadixAttention (radix tree of prefixes)
Branching agents, A/B prompts, shared system prompts
Sibling-branch KV reuse
vLLM v1
Hash-based prefix cache + PagedAttention
General serving, throughput, broad coverage
Continuous batching + prefix cache
TensorRT-LLM
Block-level KV cache with growing prefix support
Low-latency, fixed-shape, NVIDIA-only
CUDA graphs, FP8 / FP4 kernels
TGI (HuggingFace)
Has prefix caching (added 2024)
Hugging Face ecosystem deployments
Easy HF Hub integration
Triton Inference Server
No native prefix sharing (delegates to runtime)
Multi-model gateway in front of runtimes
Model orchestration, dynamic batching
Real-world examples
Real products, models, and research that use this idea.
1ChatGPT-style assistants with a long system prompt see 50%+ of their prefill cost in the system prompt; prefix caching turns that into a one-time cost amortized across all users.
2Agent frameworks (LangChain, AutoGen, custom multi-agent systems) generate many sibling branches off a planning prompt; SGLang's radix tree was designed for this branching shape.
3Anthropic's Claude API exposes explicit prompt caching with a cache_control marker; the public API surfaces what SGLang and vLLM v1 do internally.
4Coding assistants (Copilot, Cursor) often share a long file context across many completion requests in a session; prefix caching gives those a large speedup.
5Production benchmarks (e.g., LMSYS's serving comparisons) show SGLang ahead of vLLM v0 on shared-prefix workloads, with vLLM v1 closing the gap.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you estimate the cache hit rate for a production assistant before switching frameworks?
Sign in or go Pro to see the approach for this follow-up.
QRadixAttention indexes by token IDs, not text. What is the gotcha when the same logical prompt produces different token sequences?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the eviction policy in RadixAttention when the KV pool fills up?
Sign in or go Pro to see the approach for this follow-up.
QWhen would TensorRT-LLM actually be the right choice for chat?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Choosing TensorRT-LLM because it sounds the fastest. CUDA graph capture optimizes kernel launch overhead for a single fixed-shape execution; it does not deduplicate KV across two different requests that happen to share a prefix.