A semantic cache starts answering the wrong questions — locate the design flaw

The flaw is two design choices that are dangerous on their own and worse together. First, the **fixed 0.80 similarity threshold**. Cosine similarity of 0.80 is loose. Lots of query pairs that mean different things — 'cancel my order' versus 'cancel my account', or a question flipped by a single 'not' — sit above 0.80. When they do, the cache serves the nearest stored answer, which is the wrong one. That's a false hit. Second, the word **uniformly**. The bot applies this same loose rule to every intent, including billing and account deletion. Those are exactly the requests where a wrong answer causes real harm — a user told their account is gone when they only asked to pause it. The fix has two parts. Tune the threshold against labeled query pairs instead of guessing 0.80, so you can see the false-hit rate. And exclude high-stakes or fast-changing intents from semantic caching entirely, so a fuzzy match can never serve them a wrong answer.

The design has one root flaw expressed in two ways. It treats a semantic cache as a single global config — one threshold, all intents. It should instead be a per-intent risk decision. **The threshold problem.** A fixed cosine similarity of 0.80 is loose. Near-miss queries that differ by a negation, a swapped entity, or a subtly different intent routinely sit above 0.80 in embedding space. Embeddings capture topic, not the small tokens that flip meaning. Above the threshold, the cache returns the nearest stored answer — and for these pairs, that's the wrong answer served confidently. So users get answers meant for different questions. **The uniformity problem.** The same loose rule governs billing and account-deletion intents. That is what turns a quality bug into an incident. A false hit on store hours is an annoyance. A false hit on 'delete my account' versus 'pause my account' can destroy data or mislead a user about an irreversible action. The fix is two-part. Calibrate the threshold against a labeled set of equivalent and near-miss pairs so you choose it by measured false-hit rate, not intuition. And carve high-stakes or highly dynamic intents — billing, deletion, anything irreversible or time-sensitive — out of the semantic cache entirely, so they always hit the live model.