A teammate randomized system prompts per user and expects the prompt cache to still help, what's wrong?

Spot the error·Medium·4.0 · 0·~2 min·Asked atBrowserbase Coreweave·Relevant atAnthropic OpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

**Prompt caches** save money by storing the model's internal state for the leading part of a prompt that many requests share. When a new request arrives, the provider checks whether its prompt starts with the same bytes as a cached entry. If yes, it skips redoing the work for that part. The check is exact. The provider walks tokens from position zero and stops at the first token that differs. Whatever lives after that mismatch is not compared at all. The plan puts a per-user salt at the very top of the prompt. The first token is different for every user. So a request never matches another user's entry past position one, even though the next two thousand tokens are identical across users. The cache hit rate is effectively zero, not ninety percent. The second mistake is the idea that providers match prompts by meaning. They do not. Anthropic, OpenAI, and Google all key the cache on exact bytes. The fix is to move the salt out of the cacheable prefix and put it in the user message section, after the shared system prompt has been matched.

**Two compounding errors.** The argument assumes prompt caches do fuzzy matching and assumes a small leading difference is harmless. Neither is true. First error, exact prefix match. The cache walks tokens from position zero and stops at the first mismatch. A six-token salt at the head of the prompt means a request never matches another user's entry past token one. Everything after the salt, no matter how identical, is never compared. The hit rate is about zero, not ninety percent. Second error, the semantic-similarity claim. No major provider does semantic prompt caching. Anthropic, OpenAI, and Google all compare exact tokens because the cache stores deterministic KV-cache state. A fuzzy match would not be valid against a non-matching prefix. The fix is structural. Canonicalize the cacheable prefix: model identity, task instructions, tool schemas, few-shot examples. Inject all per-user data after the cache breakpoint, in the user-message section, so the system-prompt prefix stays byte-identical across users. On Anthropic the breakpoint is the cache_control marker. On OpenAI, matching on the rendered prefix is automatic, and the same rule applies. Done correctly, the hit rate climbs from zero to whatever the prefix length warrants, often eighty to ninety-five percent on a chatbot or agent loop. The diagnostic question for any prompt-caching design is the same: what are the first N tokens, and are they the same bytes across requests? If anything varies in those tokens, the cache cannot hit.

TL;DR

Prompt caches match the exact leading tokens. A per-user salt at position zero means no two users share a cached prefix, and there is no semantic fallback, so the hit rate is near 0%, not 90%.

Memory aid

Easy to grasp

Imagine a library that gives you a discount when you ask for a book whose title starts exactly the way other people's titles start. The librarian checks the first letter, then the second, then the third, and so on, until the letters stop matching. If the very first letter is your initial, nobody else's title will ever match yours past letter one. That is what putting a per-user salt at the start of a prompt does to the cache. The librarian never even gets to read the long identical part that follows, because they gave up on letter one. The fix is to keep your initial out of the title and write it on a sticker at the end, where the matching has already finished.

Key concepts

Prompt caching Prefix cache KV cache

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Prompt caching is the largest single cost saving in production LLM serving. It can cut input-token bills by an order of magnitude on workloads with a stable system prompt. Because the saving is so large, this mistake is both common and expensive. A small, well-intentioned change at the top of the prompt silently removes the entire benefit, while the team believes it is still in effect.

The plan puts a six-character per-user salt at the head of every system prompt for analytics. It predicts a ninety percent cache hit rate. Its reasons are that the rest of the prompt is identical and that providers match prompts at a semantic level. Both halves of that prediction are wrong, and they fail for different reasons rooted in how the cache works. The same pattern recurs in many forms: timestamps in the system prompt, request ids in tool schemas, per-tenant prefixes in agent loops. All of them break the cache for the same structural reason.

The cache matches the leading token sequence exactly

Hosted providers do not publish their cache internals, but Anthropic and OpenAI both document exact-prefix matching. The open-source engines show two common structures. SGLang's RadixAttention stores cached prefixes in a radix tree indexed by exact token ids, with KV-cache pages attached to the tree nodes. vLLM keys each full KV block by a hash chained from its parent block's hash, so it also behaves like a prefix tree.

A lookup walks the tree from the root. At each step it asks whether the request's next token id matches one of the children of the current node. If yes, it descends. If no, the walk stops. The node where it stops marks the longest cached prefix this request can reuse. The provider charges the discounted rate for those tokens and the full rate for everything past the stop point.

In this scenario, the first token of every request is the user-specific salt. After a few hundred requests, the root of the tree has hundreds of children, one for each unique salt token. None of them are shared across users. A request from a different user descends one step, hits no further match, and returns immediately. Only the same user's next call can continue, and only while that user's entry is still cached. The two thousand identical tokens that follow the salt are never reached because the walk already stopped at depth one.

This is not a bug in any particular provider, and you cannot tune it away with longer keys or more aggressive matching. It is the inherent behavior of any prefix-keyed structure. A single different leading token does not cost one token of cache miss. It costs the entire downstream prefix that follows that token in the tree. A cache prefix acts as a single key from position zero up to the breakpoint, and any difference along that key invalidates all of it.

Why providers cannot safely do semantic caching

The fix: cacheable prefix versus per-request tail

How to detect this bug in production

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic Claude prompt caching uses cache_control markers and matches the prefix byte-exactly up to the marker. Per-user content must live after the final marker to preserve hit rate.
OpenAI's automatic prompt caching needs the entire rendered prefix to match. On GPT-5.5, cached input costs 0.1x the standard input rate.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy can prompt caches not safely do fuzzy or semantic matching even if it were efficient?

The cache stores KV-cache state computed under specific exact tokens. Reusing it under a different leading token would feed downstream attention layers vectors that were never produced from this prompt. That breaks determinism and likely degrades output. Fuzzy matching would be incorrect, not just imprecise.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming prompt caching matches prompts by meaning or 'mostly equal' content. It does not. The lookup is a byte-exact prefix comparison from position zero, and a single different leading token invalidates the whole prefix.

60 second bullets to scan on the way to the call.

Why prefix caches match exact tokens from position zero, not semantic similarity
How a single leading-token difference invalidates the entire downstream prefix

Primary sources. Browse if you want the original framing.