You change one word in the system prompt and prompt caching breaks. Why? · Zenaique

Attempt it

Because prompt caches key on the exact sequence of token ids, not on semantic similarity. Changing one word changes the token ids from that position onward, invalidating every cached KV tensor after the change point. The KV value at position N depends on all earlier positions through attention, so once ids diverge at N, positions N+1, N+2, and so on are all stale. Even a single added space or fixed typo can shift a BPE merge boundary and cascade through the rest of the prefix.

Your answer

Prompt caching breaks because the cache keys on the exact sequence of token ids, not on what the words mean. Change one word and the token ids change from that position onward, invalidating every cached KV tensor after the change point. Here is why the damage cascades. **Prompt caches** store KV tensors for each token position. The KV value at position N was computed using attention over all positions 0 through N. If the token id at position N changes, the cached KV there is wrong. And every position after N also depended on position N through attention, so their cached values are wrong too. The entire cache from N to the end of the prefix is stale. The tokenizer makes this fragile. Even a single added space or fixed typo can shift a BPE merge boundary, changing the token id at one position and cascading through every subsequent position. Two prompts that look nearly identical to a human can produce completely different token id sequences from the edit point forward. The practical defense: keep the cacheable prefix byte-identical across calls. Put stable content (system prompt, tool schemas) at the front and volatile content (user message, timestamps) at the end. The most common cache-destroying bug is interpolating a timestamp or request id near the start of the system prompt.

**Prompt caches** store the KV tensors for a prefix at the token-id level. The cache key is the exact sequence of token ids; two requests share a cache hit only up to the first position where their ids differ. This is why even a one-character change can reset the cache. The tokenizer is byte-sensitive: adding a space, fixing a typo, or reordering a sentence usually shifts a token boundary, and ids from that point onward differ. Once a single id differs at position N, the cached KV at N is wrong (computed for the old id), and every position after N depended on N through attention, so all are wrong too. The model recomputes from N to the end. Providers implement this differently but the underlying mechanic is the same. Anthropic uses explicit `cache_control` breakpoints; cached input bills at roughly one-tenth normal rate. OpenAI applies automatic prefix caching above a minimum length and reports cached tokens in usage. Google Gemini offers explicit context caching with TTL. vLLM and SGLang implement prefix caching as kernel-level KV reuse keyed on the token-id prefix. The design rule: order the prompt with stable content first (system instructions, tool schemas, retrieved knowledge) and volatile content last (user message, timestamps). Keep the cacheable prefix byte-identical across calls. The single most common bug is interpolating a timestamp or request id near the start of the system prompt, which makes every call a cache miss.

Quick reference

TL;DR

Caches key on exact token ids, not meaning. One changed word shifts ids from that point forward, invalidating every KV tensor after it because attention at position N depends on all earlier positions.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a long shared road the model drives down for every request. Once the model has driven a mile, the road behind has cone markers showing it has been measured. The next driver can skip the measured section. Now suppose someone moves a single cone earlier in the road. Every cone past that point is suspicious; the next driver has to re-measure from the moved cone onward. The cones are the cached KV values, the road is the token prefix, and moving a cone is what happens when you change a single character in the prompt.

Key concepts

Prompt caching KV cache Prefix cache

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: explain the cache as a token-id prefix store, walk through why a single id change invalidates everything after it, contrast Anthropic explicit breakpoints with OpenAI automatic caching, and close on the stable-prefix volatile-suffix design rule.

Real-world examples

Real products, models, and research that use this idea.

Anthropic Claude API: cache_control markers in message blocks; cached input billed at roughly one-tenth of normal input price in 2026.
OpenAI prompt caching: automatic for prompts above a minimum token length on the GPT-5.5 family; usage.prompt_tokens_details.cached_tokens reports hits.
Google Gemini context caching: explicit cache objects with TTL, billed at a reduced rate.
vLLM and SGLang prefix caching: kernel-level KV reuse keyed on the token-id prefix, enabled by default in 2026 serving.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf I edit a single word in the middle of a 10K token system prompt, how much of the cache is invalidated?

Sign in or go Pro to see the approach for this follow-up.

QWhy does retrieval order affect cache hits in a RAG application?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Putting dynamic content (user question, timestamp) at the start of the prompt and the long static system prompt at the end. The first token differs every call, so the cache hit rate drops to zero.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.