Find the flaw: this semantic answer cache started serving answers to the wrong questions
Click any words you think contain an error. Click again to unmark.
The bug is the cosine threshold of 0.75 — too loose for a semantic answer cache. Distinct questions sharing topic words collide above it, so it serves wrong answers. A tight bar near 0.95 admits only true paraphrases.
Imagine a help desk that keeps a folder of past answers. When a new question comes in, the clerk checks if it 'feels similar' to one already answered and, if so, hands over that old answer instead of doing the work again. The trouble is the clerk's bar for 'similar' is way too low. 'How do I reset my password' and 'how do I change my password' feel similar — they share most of the same words — but they want different answers. With a loose bar, the clerk confidently hands over the wrong one. The fix is to make the clerk much stricter, only reusing an old answer when the new question is nearly word for word the same.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Semantic answer caching is one of those production optimizations that looks trivially simple and hides a sharp edge. The idea is irresistible: LLM calls are slow and expensive, so when a question is "the same" as one you have already answered, just return the stored answer. Embed the query, find the nearest cached query, and if they are close enough, reuse the answer.
The entire system hinges on what "close enough" means, and this snippet gets it wrong with a single number. The cosine bar of 0.75 feels reasonable to anyone who has tuned a retriever, which is exactly why it is a great spot the error: the mistake is importing a correct instinct from the wrong context.
This deep dive explains why a cache demands the opposite precision-recall tradeoff from retrieval, why topically similar questions collide at a loose threshold, and what a correct semantic cache needs beyond just a higher number.
Where the snippet goes wrong
Walk the code. It embeds the incoming query, finds the nearest cached query vector, and if cosine(q_vec, hit.vec) > 0.75 it returns the cached answer. Otherwise it runs the full RAG pipeline and stores the new result.
The structure is fine. The bug is the constant 0.75. It is the single gate that decides whether two questions count as the same, and it is set far too low for this job.
A cosine of 0.75 between two query embeddings does not mean the questions are equivalent. It means they are broadly in the same neighborhood — same topic, overlapping words, related intent. That is a perfectly good bar for deciding whether a document is worth retrieving. It is a terrible bar for deciding whether you can hand a user an answer written for a different question.
The correction raises the threshold to roughly 0.95, where only near-identical paraphrases clear it. Everything else falls through to the pipeline. The rest of this explanation is about why that gap between 0.75 and 0.95 is the difference between a useful cache and one that lies to users.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- GPTCache and similar semantic-cache libraries expose a similarity threshold; setting it too low causes cross-question collisions in production.
- Support assistants caching 'reset password' answers and serving them to 'change password' queries because both exceed a loose cosine bar.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you actually choose the cache threshold rather than guessing 0.95?
Build a labeled set of true-paraphrase and near-miss query pairs, plot precision against the threshold, and pick the point where false-hit precision meets your tolerance.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating cosine similarity as a calibrated quality score and picking a low threshold like 0.75 the way you would for retrieval recall — caching needs precision, so the bar has to be near-identical, not just topically close.
60 second bullets to scan on the way to the call.
Why a semantic answer cache needs precision, not recall
How a false cache hit differs from a cache miss in user impact
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.