Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
**Prompt caches** save money by storing the model's internal state for the leading part of a prompt that many requests share. When a new request arrives, the provider checks whether its prompt starts with the same bytes as a cached entry. If yes, it skips redoing the work for that part.
The check is exact. The provider walks tokens from position zero and stops at the first token that differs. Whatever lives after that mismatch is not compared at all, because there is no point.
The plan in the question puts a per-user salt at the very top of the prompt. The first token is different for every user. So the cache walk stops at position one for every request, even though the next two thousand tokens are identical across users. The cache hit rate is effectively zero, not ninety percent.
The second mistake is the idea that providers match prompts by meaning. They do not. Anthropic, OpenAI, and Google all key the cache on exact bytes. The fix is simple: move the salt out of the cacheable prefix and put it in the user message section, after the shared system prompt has been matched.
**Two compounding errors.** The argument assumes prompt caches do fuzzy matching and assumes a small leading difference is forgivable. Neither is true.
First error, exact prefix match. The cache walks tokens from position zero and stops at the first mismatch. A six-token salt at the head of the prompt means the walk stops at token one. Everything after the salt, no matter how identical, is never compared. The hit rate is approximately zero, not ninety percent.
Second error, the semantic-similarity claim. No major provider does semantic prompt caching. Anthropic, OpenAI, and Google all compare exact tokens because the cache stores deterministic KV-cache state. A fuzzy match would not be valid against a non-matching prefix.
The fix is structural. Canonicalize the cacheable prefix: model identity, task instructions, tool schemas, few-shot examples. Inject all per-user data after the cache breakpoint, in the user-message section, so the system-prompt prefix stays byte-identical across users. On Anthropic this is the cache_control marker; on OpenAI the prefix is implicit but the same discipline applies. Done correctly, the hit rate climbs from zero to whatever the prefix length warrants, often eighty to ninety-five percent on a chatbot or agent loop.
The diagnostic question for any prompt-caching design is the same: what are the first N tokens, and are they truly the same bytes across requests? If anything varies in those tokens, the cache is dead.
Quick reference
TL;DR
Prompt caches are exact-prefix matchers keyed on the literal leading token bytes; a six-token salt at position zero invalidates the entire cached prefix and there is no semantic fallback, so the expected ~90% hit
Imagine a library that gives you a discount when you ask for a book whose title starts exactly the way other people's titles start. The librarian checks the first letter, then the second, then the third, and so on, until the letters stop matching. If the very first letter is your initial, nobody else's title will ever match yours past letter one. That is what putting a per-user salt at the start of a prompt does to the cache. The librarian never even gets to read the long identical part that follows, because they gave up on letter one. The fix is to keep your initial out of the title and write it on a sticker at the end, where the matching has already finished.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Workloads where every prompt is genuinely unique end to end If there is no shared prefix across calls, prompt caching cannot help no matter where the salt sits. The discussion only matters when a stable prefix exists to protect.
Models or providers that do not offer prompt caching at all Some smaller or self-hosted setups skip the feature entirely, so prefix discipline is moot for them. Re-evaluate when you migrate to a caching-capable provider.
Single-turn applications with very short system prompts If the cacheable prefix is under the provider's minimum (often ~1024 tokens on OpenAI), the cache never engages and the salt placement does not change costs.
Workloads where per-user behavior must influence the system prompt itself If user attributes truly belong in the persona or instructions, you trade cache hit rate for personalization. That is a legitimate choice as long as the tradeoff is explicit.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name exact-prefix matching, walk the trie lookup, debunk the semantic-similarity claim, and prescribe moving per-user data after the cache breakpoint.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic Claude prompt caching uses cache_control markers and matches the prefix byte-exactly up to the marker; per-user content must live after the final marker to preserve hit rate.
2OpenAI's automatic prompt caching keys on the leading 1024+ tokens of the request and offers a ~50% input-token discount on hits in 2026 pricing.
3Google Gemini's context caching API similarly requires explicit cached-content blocks that are matched by id and content, never by similarity.
4vLLM's RadixAttention prefix cache is implemented as a radix tree on token ids, making the 'first token differs = full miss' behavior structural rather than a policy choice.
5Production debugging on agent frameworks like LangGraph and Anthropic's Computer Use SDK regularly catches this bug by logging prefix-cache hit rate per turn and noticing 0% hits where 90% was expected.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy can prompt caches not safely do fuzzy or semantic matching even if it were efficient?
Sign in or go Pro to see the approach for this follow-up.
QHow should you meet a per-user analytics requirement without breaking the prompt cache?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right way to measure realized prompt cache hit rate to catch this bug in production?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming prompt caching matches prompts by meaning or 'mostly equal' content. It does not. The lookup is a byte-exact prefix comparison from position zero, and a single different leading token invalidates the whole prefix.