In agent and chatbot fleets, many concurrent requests share a long system prompt or few-shot prefix. Explain how prefix KV-cache sharing reduces serving cost. Cover what is shared, what remains per request, and which serving primitive makes it practical.
Prefill the shared prefix once, alias its KV pages across every request that starts with the same tokens, and per-request work begins only at the divergent suffix.
Imagine a busy bakery where every customer orders a custom cake but they all start with the same vanilla base. A wasteful baker mixes a fresh batch of vanilla base for every order. A smart baker mixes one big batch of base in the morning and scoops from it for every order, only doing the custom decoration per customer. Prefix KV-cache sharing is the smart baker for LLM serving: the long system prompt is the shared vanilla base, and only the user's unique question gets fresh work. The bakery serves the same customers, faster, with less flour.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prefix KV-cache sharing is the single biggest serving-cost optimization for modern agent and chatbot workloads. The pattern is everywhere: every user request carries a long shared system prompt (tool definitions, persona, few-shot examples) and a short divergent suffix (the actual user message). Naive serving recomputes the prefix's prefill work for every request; prefix sharing makes it happen once.
This card walks the mechanism end to end. We start with the cost being attacked, prefill compute and cache memory that scale linearly with concurrent requests despite the prefix being byte-identical. We then walk the hash-keyed page lookup that detects sharing opportunities, the refcounting that keeps shared pages alive across requests, and the paged KV allocation primitive that makes the whole thing practical.
By the end you should be able to reason about which workloads benefit, which break the mechanism, and how prefix sharing composes with the other 2026-production-stack cache optimizations (PagedAttention, GQA, KV quantization, continuous batching).
What gets duplicated in naive serving
A modern agent prompt typically looks like this:
[2000-token shared system prompt: tool definitions, persona, examples]
[50-token user message]
The shared portion is byte-identical across thousands of concurrent users. Naive serving runs full prefill on the entire prompt per request: forward pass over all 2050 tokens, write K and V at every layer to that request's KV cache buffer.
What scales linearly
- Prefill compute: attention is
O(n^2 d)per layer, FFN isO(n d^2)per layer. For 2000-token prefixes on Llama 4 Maverick, that is gigaflops per request that nothing depends on except the prefix tokens themselves. - Cache memory: each request's KV cache holds the prefix's K and V at every layer, every head. For Llama-3 70B GQA-8 fp16 at 2000 tokens, that is 5.24 MB per block times 125 blocks, roughly 650 MB per request.
With 100 concurrent users, the same 650 MB of prefix K, V exists 100 times in HBM. Every byte of that is pure waste, the values are identical.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Conceptual prefix-sharing lookup at request arrival.
# Real implementations (vLLM, SGLang) integrate this into the scheduler.
BLOCK_SIZE = 16
class PrefixCache:
def __init__(self):
self.block_table = {} # hash -> physical_block_id
self.refcounts = {}
def lookup_prefix(self, tokens):
"""Return list of (physical_block_id, hit_or_miss) per block."""
page_table = []
for i in range(0, len(tokens), BLOCK_SIZE):
block = tuple(tokens[i:i + BLOCK_SIZE])
h = hash(block)
if h in self.block_table:
phys = self.block_table[h]
self.refcounts[phys] += 1
page_table.append((phys, 'hit'))
else:
phys = allocate_new_block()
self.block_table[h] = phys
self.refcounts[phys] = 1
page_table.append((phys, 'miss'))
# Subsequent blocks must also be miss (suffix divergence)
break
return page_table
def release(self, phys):
self.refcounts[phys] -= 1
if self.refcounts[phys] == 0:
free_block(phys)Real products, models, and research that use this idea.
- vLLM's automatic prefix caching is the default in production deployments serving Llama 4 Maverick and Qwen 3.5 across long system prompt workloads.
- Anthropic's prompt caching feature for Claude Opus 4.7 lets users mark a prefix as cacheable and bills cached tokens at ~10% of the standard input rate.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can't you share KV pages across requests where the prefix is at different absolute positions?
Modern position encodings (RoPE, sinusoidal, ALiBi) bake position into either the K, Q values or the attention bias. K computed at position 5 with rotation angle theta_5 is not the same tensor as K computed at position 12. Cached pages are valid only at the position they were computed for, which usually means absolute position zero in the request.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming prefix sharing requires similar prompts. It requires byte-identical prefix tokens. One word change at position zero invalidates every downstream cache page because K and V at later positions depend on the full prefix.
60 second bullets to scan on the way to the call.
What prefill cost is, attention plus FFN over the prompt before decode starts
Why a 2000-token shared system prompt is the canonical target
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.