Click any words you think contain an error. Click again to unmark.
Putting a per-request timestamp at the very front changes the first bytes of every prompt, so the exact-prefix cache never hits — move volatile content after the stable prefix.
Imagine a library where they pre-process a thick reference book once so anyone can look things up fast. But someone insists on writing today's date on the very first page before each reader opens it. Now the first page is different every time, so the librarian thinks it's a brand-new book and re-processes the whole thing from scratch. That's what's happening here. The cache reuses work only when the start of the prompt is exactly the same. A timestamp stamped at the top changes the start on every request, so nothing matches and the cache is useless. The fix isn't to drop the timestamp — it's to move it to the end, after the parts that never change.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This scenario looks like a caching bug and is really a prompt-layout bug. The team did the hard part right — they enabled caching and they have a genuinely cacheable workload, with fixed system instructions and a shared document repeated across requests. Yet their hit rate is zero, which feels like the feature is broken.
It isn't broken; it's being fed prompts it can't match. The whole behavior follows from one property of prefix caching that's easy to forget under deadline pressure: the cache matches from the very first token, and the match must be exact. A single dynamic token at the front is enough to defeat it entirely, no matter how stable everything behind that token is.
This walkthrough traces the failure step by step — what the cache compares, why a leading timestamp forces an immediate miss, and why the large stable body gets stranded. Then it gives the fix as a layout principle, generalizes it to every other source of per-request entropy, and closes on why this class of bug is dangerous specifically because it fails silently rather than throwing an error.
What the cache compares, and where it gives up
Prefix caching reuses the prefill KV state of a leading segment of the prompt. To decide what it can reuse, the provider compares the incoming prompt against previously cached prefixes starting at token zero. It walks forward, reusing cached state token by token, and stops at the first position where the current prompt differs from the cached one. Everything before that divergence is a cache hit; everything from the divergence onward is re-prefilled fresh.
The critical word is exact. The comparison is byte for byte on tokens, not fuzzy or semantic. "Almost the same prompt" reuses nothing past the first differing token. So the length of the reusable span is entirely determined by how far into the prompt the first difference appears.
That reframes the whole design goal. To get good cache utilization, you want the first point of difference between any two requests to come as late as possible. Equivalently, you want the longest possible run of identical leading tokens shared across requests. Anything that introduces a difference early — at or near token zero — caps the reusable span at almost nothing, regardless of how much identical content sits further back. The cache can't skip ahead to find the stable part; it only ever reuses an unbroken run from the start.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt-caching docs warn that any content before a cache breakpoint must be identical or the cache misses.
- OpenAI automatic prompt caching matches on the exact prompt prefix, so a leading dynamic token drops the hit rate to zero.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect this regression in production before the bill spikes?
Track cached input token counts and cache hit rate in usage metrics; a near-zero rate flags a front-loaded dynamic field.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Concluding the cache is broken or disabled, when really a dynamic value at position zero changes the prefix every call so the exact-prefix match never fires.
60 second bullets to scan on the way to the call.
Why prefix caching matches from the very first token
How a leading dynamic value forces divergence at position zero
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.