A teammate randomized system prompts per user and expects the prompt cache to still help, what's wrong?

Spot the error·Medium·4.0 · 0·~2 min·Asked atBrowserbase Cognizant Coreweave·Relevant atAnthropic OpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

**Prompt caches** save money by storing the model's internal state for the leading part of a prompt that many requests share. When a new request arrives, the provider checks whether its prompt starts with the same bytes as a cached entry. If yes, it skips redoing the work for that part. The check is exact. The provider walks tokens from position zero and stops at the first token that differs. Whatever lives after that mismatch is not compared at all, because there is no point. The plan in the question puts a per-user salt at the very top of the prompt. The first token is different for every user. So the cache walk stops at position one for every request, even though the next two thousand tokens are identical across users. The cache hit rate is effectively zero, not ninety percent. The second mistake is the idea that providers match prompts by meaning. They do not. Anthropic, OpenAI, and Google all key the cache on exact bytes. The fix is simple: move the salt out of the cacheable prefix and put it in the user message section, after the shared system prompt has been matched.

**Two compounding errors.** The argument assumes prompt caches do fuzzy matching and assumes a small leading difference is forgivable. Neither is true. First error, exact prefix match. The cache walks tokens from position zero and stops at the first mismatch. A six-token salt at the head of the prompt means the walk stops at token one. Everything after the salt, no matter how identical, is never compared. The hit rate is approximately zero, not ninety percent. Second error, the semantic-similarity claim. No major provider does semantic prompt caching. Anthropic, OpenAI, and Google all compare exact tokens because the cache stores deterministic KV-cache state. A fuzzy match would not be valid against a non-matching prefix. The fix is structural. Canonicalize the cacheable prefix: model identity, task instructions, tool schemas, few-shot examples. Inject all per-user data after the cache breakpoint, in the user-message section, so the system-prompt prefix stays byte-identical across users. On Anthropic this is the cache_control marker; on OpenAI the prefix is implicit but the same discipline applies. Done correctly, the hit rate climbs from zero to whatever the prefix length warrants, often eighty to ninety-five percent on a chatbot or agent loop. The diagnostic question for any prompt-caching design is the same: what are the first N tokens, and are they truly the same bytes across requests? If anything varies in those tokens, the cache is dead.

TL;DR

Prompt caches are exact-prefix matchers keyed on the literal leading token bytes; a six-token salt at position zero invalidates the entire cached prefix and there is no semantic fallback, so the expected ~90% hit

Memory aid

Easy to grasp

Imagine a library that gives you a discount when you ask for a book whose title starts exactly the way other people's titles start. The librarian checks the first letter, then the second, then the third, and so on, until the letters stop matching. If the very first letter is your initial, nobody else's title will ever match yours past letter one. That is what putting a per-user salt at the start of a prompt does to the cache. The librarian never even gets to read the long identical part that follows, because they gave up on letter one. The fix is to keep your initial out of the title and write it on a sticker at the end, where the matching has already finished.

Key concepts

Prompt caching Prefix cache KV cache

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Prompt caching is the single largest cost lever in production LLM serving in 2026, often cutting input-token bills by an order of magnitude on workloads with a stable system prompt. Because the win is so large, the mistake in front of us is both common and expensive: a small, well-intentioned change at the top of the prompt silently destroys the entire benefit while the team believes it is still in effect.

The author proposes putting a six-character per-user salt at the head of every system prompt for analytics, and predicts a ninety percent cache hit rate on the basis that the rest of the prompt is identical and providers match prompts at a semantic level. Both halves of that prediction are wrong, and they fail for different reasons rooted in how the cache actually works. The error is worth dissecting in detail because the same pattern recurs in many forms: timestamps in the system prompt, request ids in tool schemas, per-tenant prefixes in agent loops. All of them break the cache for the same structural reason.

This walkthrough builds up the corrected mental model in three steps. First, the data structure: what is the cache actually checking, and why does that make leading content uniquely toxic. Second, the determinism argument: why no provider can safely do fuzzy matching even if they wanted to. Third, the fix pattern: where to place per-user data so the cache survives and the analytics goal is still met. By the end you should be able to predict the realized hit rate of any prompt design by inspection, and to redesign a broken one in under a minute.

The cache is a token-trie keyed on the leading sequence

Every production prompt cache is implemented as some flavor of token-prefix tree. Anthropic and OpenAI keep the internals opaque, but the open-source implementations make the structure explicit: vLLM's RadixAttention and SGLang's prefix-cache module both store the leading tokens of every served request in a radix tree, indexed by the exact token ids, with KV-cache blocks attached to each tree node.

A lookup walks the tree from the root. At each step it asks whether the request's next token id matches one of the children of the current node. If yes, it descends. If no, the walk stops. Whatever node it stopped at marks the longest cached prefix this request can reuse, and the provider charges the discounted rate for those tokens and the full rate for everything past the stop point.

The consequence for our scenario is unambiguous. The first token of every request is the user-specific salt. The root of the tree has, after a few hundred requests, hundreds of children, one for each unique salt token. None of them are shared across users. Every walk descends one step, hits no further match, and returns immediately. The two thousand identical tokens that follow the salt are never reached because the walk already stopped at depth one.

This is not a bug in any particular provider, and you cannot tune it away with longer keys or more aggressive matching. It is the inherent behavior of any prefix-keyed structure. A single different leading token does not cost one token of cache miss, it costs the entire downstream prefix that lived behind that token in the tree. The right intuition is that a cache prefix is a single atomic key from position zero up to the breakpoint, and any difference along that key invalidates the whole atom.

Why providers cannot safely do semantic caching

The fix: cacheable prefix versus per-request tail

How to detect this bug in flight

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic Claude prompt caching uses cache_control markers and matches the prefix byte-exactly up to the marker; per-user content must live after the final marker to preserve hit rate.
OpenAI's automatic prompt caching keys on the leading 1024+ tokens of the request and offers a ~50% input-token discount on hits in 2026 pricing.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can prompt caches not safely do fuzzy or semantic matching even if it were efficient?

The cache stores KV-cache state computed under specific exact tokens. Reusing it under a different leading token would feed downstream attention layers vectors that were never produced from this prompt, breaking determinism and likely degrading output. Fuzzy matching would be incorrect, not just imprecise.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming prompt caching matches prompts by meaning or 'mostly equal' content. It does not. The lookup is a byte-exact prefix comparison from position zero, and a single different leading token invalidates the whole prefix.

60 second bullets to scan on the way to the call.

Why prefix caches match exact tokens from position zero, not semantic similarity
How a single leading-token difference invalidates the entire downstream prefix

Primary sources. Browse if you want the original framing.