Heavy multi-turn chat shares system prompts across requests, pick the serving framework that exploits that best.

Multi-turn chat has a specific cost structure. The first message in a conversation usually starts with a long system prompt (say, 2k tokens of instructions and persona), then a series of user and assistant turns. Every time a new turn arrives, the model has to process the **entire conversation history** as the input prompt, prefill its KV cache, and only then start generating the next response. If you have many parallel users sharing the same system prompt (which is the production reality, every API user of a given assistant sees the same system prompt), the model is recomputing the **same KV cache** for that prefix again and again. This is wasted work. The fix is **prefix caching**: store the KV cache for a prefix once, and reuse it whenever a new request shares that prefix. Two frameworks lead on this: vLLM v1 (general prefix caching) and **SGLang** (RadixAttention, which organizes prefixes in a radix tree to handle deep branching). The other three options describe real but unrelated optimizations: - **TensorRT-LLM**: CUDA graph capture, which speeds up fixed-shape execution by replaying recorded kernel launches. It is not about KV reuse across requests. - **Triton Inference Server**: dynamic batching merges concurrent requests for batched execution. It is not about prefix sharing. - **Hugging Face TGI**: tokenizers do not decide whether you recompute KV; that is a confusion of layers. When the workload is 'lots of conversations sharing a system prompt and branching from there', SGLang's radix structure is the natural fit.

Multi-turn chat is the canonical workload for **prefix-aware KV reuse**, and the question turns on whether you can identify which framework has a first-class data structure for that. **SGLang's RadixAttention** is a radix tree (a compressed trie) keyed by token IDs. Every prefix that has been seen has a node, and the KV blocks for that prefix live behind that node. When a new request comes in, the scheduler walks the tree to find the longest matching prefix and resumes from there. Two requests sharing a prefix of length P pay the prefill cost for P only once. The **radix tree is the differentiator** from vLLM's prefix caching. vLLM v1 caches prefixes well in the common case (single prefix per system prompt), but RadixAttention shines when conversations **branch**: an agent forks the same system prompt into 10 sub-tasks, an A/B test sends two variants of the same prompt, a structured generator explores many continuations. The radix structure naturally handles sibling branches at any depth. **TensorRT-LLM** uses CUDA graphs to amortize kernel launch overhead, which matters for low-latency single-request decode but is orthogonal to KV reuse across requests. **Triton Inference Server** (NVIDIA's general inference server) does dynamic batching, which merges concurrent requests for batched throughput, not for prefix dedup. **TGI** is solid for production deployment but its tokenizer is a tokenizer; nothing about a tokenizer decides whether you recompute KV. So the framework choice maps directly to the workload's shape. For heavy shared-prefix multi-turn chat, RadixAttention is built for the case. vLLM v1 is the strong runner-up.