Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
The mistake is **where the timestamp sits**, not that the team uses caching wrong in some exotic way.
Prefix caching reuses work only when the beginning of the prompt is identical from one request to the next. It compares prompts from the very first token and reuses everything up to the point where they differ.
Here the team puts a fresh timestamp at the very top of every prompt. Since the timestamp changes on every call, the first tokens are different every time. The cache compares prompts, sees they diverge immediately at position zero, and finds nothing to reuse. The hit rate drops to zero.
The stable, cacheable content — the fixed system instructions and the shared document — is sitting behind the timestamp, so it can never form a reusable prefix.
The fix is just ordering. Put the stable content first: system instructions, then the shared document. Put anything that changes per request — the timestamp, the user ID, the live question — after that stable block. Now the long stable head is identical across calls, the cache hits, and the timestamp still gets logged.
The error is prompt **ordering**: a per-request timestamp is placed before the stable content, which poisons the cacheable prefix.
Prefix caching keys on an exact, stable leading segment. The provider walks the prompt from token zero and reuses cached KV state up to the first position where this request diverges from a previously cached one. Anything dynamic at the front moves that divergence point to the very start.
In this design the timestamp changes on every request, so byte zero already differs. No two prompts share a leading prefix, so the match fails immediately and the hit rate collapses to essentially zero. The system instructions and shared document behind it are completely stable and would otherwise cache beautifully.
The fix is to reorder, not to remove. Sort content stable to volatile. First come the fixed system instructions. Next comes the shared knowledge document. Last come the volatile per-request fields: the timestamp, user ID, and the live query.
Now the long stable head is byte for byte identical across calls, so it caches, and the dynamic suffix is the only part re-prefilled each time. The team keeps their debug timestamp; it just lives after the cached prefix instead of in front of it. This single reordering can take the hit rate from zero to near the share of the prompt that's stable.
Quick reference
TL;DR
Putting a per-request timestamp at the very front changes the first bytes of every prompt, so the exact-prefix cache never hits — move volatile content after the stable prefix.
Imagine a library where they pre-process a thick reference book once so anyone can look things up fast. But someone insists on writing today's date on the very first page before each reader opens it. Now the first page is different every time, so the librarian thinks it's a brand-new book and re-processes the whole thing from scratch.
That's what's happening here. The cache reuses work only when the start of the prompt is exactly the same. A timestamp stamped at the top changes the start on every request, so nothing matches and the cache is useless. The fix isn't to drop the timestamp — it's to move it to the end, after the parts that never change.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A prompt that genuinely must start with per-request content for correctness If the live query truly belongs first, caching the prefix is impossible, so reorder only when the model tolerates it.
Workloads where the stable head is tiny relative to the dynamic part Reordering buys little when there's barely any stable prefix to cache, so the gain may not justify the change.
Sparse traffic where cache entries expire between requests Even perfectly ordered prompts miss if the TTL lapses, so ordering alone won't restore hit rate at low volume.
Quick scan · 2–4 min
Everything important, quickly.
Take 5-7 minutes to explain exact-prefix matching, why a leading timestamp forces a position-zero miss, and the stable to volatile reordering fix.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's prompt-caching docs warn that any content before a cache breakpoint must be identical or the cache misses.
2OpenAI automatic prompt caching matches on the exact prompt prefix, so a leading dynamic token drops the hit rate to zero.
3Production teams watch cached-token counts in usage dashboards to catch a front-loaded dynamic field killing reuse.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect this regression in production before the bill spikes?
Sign in or go Pro to see the approach for this follow-up.
QWhere exactly should the live user query go relative to the cached prefix?
Sign in or go Pro to see the approach for this follow-up.
QWhat other per-request fields commonly poison the prefix the same way?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Concluding the cache is broken or disabled, when really a dynamic value at position zero changes the prefix every call so the exact-prefix match never fires.