Same topic, related formats. Practice these next.
In agent and chatbot fleets, many concurrent requests share a long system prompt or few-shot prefix. Explain how prefix KV-cache sharing reduces serving cost. Cover what is shared, what remains per-request, and which serving primitive makes it practical.
Prefill the shared prefix once, alias its KV pages across every request that starts with the same tokens, and per-request work begins only at the divergent suffix.
Imagine a busy bakery where every customer orders a custom cake but they all start with the same vanilla base. A wasteful baker mixes a fresh batch of vanilla base for every order. A smart baker mixes one big batch of base in the morning and scoops from it for every order, only doing the custom decoration per customer. Prefix KV-cache sharing is the smart baker for LLM serving: the long system prompt is the shared vanilla base, and only the user's unique question gets fresh work. The bakery serves the same customers, faster, with less flour.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
8 min: workload pattern (shared system prompts), prefill cost economics, page hashing and aliasing mechanism, what stays per-request, paged KV as the enabling primitive, real-world pricing signals from Anthropic and vLLM.
# Conceptual prefix-sharing lookup at request arrival.
# Real implementations (vLLM, SGLang) integrate this into the scheduler.
BLOCK_SIZE = 16
class PrefixCache:
def __init__(self):
self.block_table = {} # hash -> physical_block_id
self.refcounts = {}
def lookup_prefix(self, tokens):
"""Return list of (physical_block_id, hit_or_miss) per block."""
page_table = []
for i in range(0, len(tokens), BLOCK_SIZE):
block = tuple(tokens[i:i + BLOCK_SIZE])
h = hash(block)
if h in self.block_table:
phys = self.block_table[h]
self.refcounts[phys] += 1
page_table.append((phys, 'hit'))
else:
phys = allocate_new_block()
self.block_table[h] = phys
self.refcounts[phys] = 1
page_table.append((phys, 'miss'))
# Subsequent blocks must also be miss (suffix divergence)
break
return page_table
def release(self, phys):
self.refcounts[phys] -= 1
if self.refcounts[phys] == 0:
free_block(phys)Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming prefix sharing requires similar prompts. It requires byte-identical prefix tokens. One word change at position zero invalidates every downstream cache page because K and V at later positions depend on the full prefix.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.