Order the cache layers an LLM request should check before paying for tokens
- 1Normalize the request: strip volatile fields like timestamps and request IDs to produce a stable cache key
- 2Call the model, then write the fresh response back into the response cache with a TTL
- 3Check the semantic cache: embed the query and look for a previously answered near duplicate above the similarity threshold
- 4Check the exact match response cache; on a hit, return the stored answer at near zero cost
- 5On a full miss, send the request with the stable system prompt first so the provider's prompt prefix cache can reuse prefill work
Normalize first to get a stable key, check exact-match, then semantic, then structure the prompt so the provider's prefix cache helps, then call the model and write back.
Imagine you are answering homework questions for a class. Before you do any work, you check if the same question is already answered on the noticeboard. If it is, you read it out and you are done. If it is not, you check if a very similar question has been answered, and if it is close enough you read that one out. Only if neither helps do you actually sit down and work it out. And when you do, you write the new answer on the noticeboard so the next student does not need to ask you again. The trick is the order: the more expensive the step, the further back in the line you put it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM tokens are the most expensive part of most LLM products by a comfortable margin, and the cheapest token is the one you do not have to generate. Caching is therefore one of the highest-leverage tools in LLM system design, and the order in which you check caches matters as much as having them at all.
This sequence is a triage hierarchy: each layer is more expensive than the one before, so checking in cost order means a request that could have been answered by the cheapest layer never reaches the costliest one. Misorder the layers and you spend money on lookups you did not need.
The deep dive walks each layer, names the failure mode that makes it look worse than it is, and finishes with the cache people forget: the provider's prompt-prefix cache that lives on the model side of the wire.
Layer zero: normalization, the cache that does not exist
Normalization is the cache key contract. Real LLM requests carry a surprising amount of debris that should not affect the answer: a request ID for tracing, a timestamp the client added, a session token, a user identifier spliced into a template for personalization, optional fields in different orders depending on which client library produced them. Hash the raw request and every one of these knocks the cache key around, so two semantically identical requests look completely different to a key-value store.
The normalization pass strips volatile fields by allowlist (only the fields you want to participate in the key survive), sorts dict keys into a canonical order, normalizes whitespace, and lower-cases where casing is irrelevant. The output is a deterministic string, hashed once, used as the key for the exact-match cache. The discipline is that any field with no semantic effect on the answer must not enter the key, and any field with semantic effect must.
Teams that report a one-percent cache hit rate are usually not running out of duplicates; they have broken their key by passing a timestamp or request ID through. Fix the normalization step and the hit rate jumps. This is unglamorous work and the most common silent failure of an LLM cache.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt caching on Claude Opus 4.7 and Sonnet 4.6 charges discounted rates for cached prompt prefixes, which is exactly the layer-four win the order step captures.
- OpenAI prompt caching on GPT-5.5 automatically reuses prefixes of at least a configured token length, so leaving the system prompt at the start of the request is the entire opt-in.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you tune the semantic-cache similarity threshold?
Build a labeled set of paraphrase pairs and near-miss adversarial pairs, sweep the threshold, and pick the point that maximizes recall at an acceptable false-positive rate.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping normalization and getting near-zero hit rates because timestamps and request IDs poisoned the key, or putting the semantic cache before the exact cache and paying for an embedding lookup on every request.
60 second bullets to scan on the way to the call.
What does request normalization do, and why is it the prerequisite to any cache key?
Why is exact-match cheaper than semantic cache, and what does each one catch?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.