Click any words you think contain an error. Click again to unmark.
A fixed 0.80 threshold applied uniformly to every intent lets near-miss queries collide onto wrong cached answers — and routes billing and deletion requests through a fuzzy cache that can do real harm.
Imagine a clerk who hands you a pre-written answer card whenever your question 'sounds about 80% like' one they've seen. For 'what are your store hours?' that's harmless if they're slightly off. But they use the same loose rule for 'delete my account' and 'pause my account' — which sound alike but mean opposite things. So someone asking to pause their account gets the 'we deleted it' card. The mistake is one loose 'sounds-alike' rule used for everything, including the requests where a wrong answer is a disaster.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This scenario is a realistic post-mortem in miniature. Someone shipped a semantic cache to cut inference cost on a support bot, picked a similarity threshold that felt reasonable, and switched it on for all traffic. Then users started getting answers to questions they didn't ask. The job is to locate the design flaw — and the instructive part is that there are two, and they compound.
The surface symptom is false hits: the cache serving a stored answer that belongs to a different question. But the root cause isn't a typo or a bug in the retrieval code. It's a conceptual error about what a semantic cache is. The builder treated it as a piece of infrastructure with one global setting, when it's actually a per-intent correctness decision dressed up as a cache.
This deep dive separates the two flaws cleanly — why 0.80 is a loose threshold rather than a safe one, and why 'uniformly' is an independent and arguably worse mistake — then builds the risk-tiered caching policy that fixes both, and shows how to verify the fix with a labeled set and production monitoring.
Flaw one: 0.80 is loose, not safe
The first instinct on seeing 'cosine similarity exceeds 0.80' is that 0.80 sounds high — 80%, surely that's a strict match. That instinct is exactly the trap.
Cosine similarity between sentence embeddings doesn't behave like a percentage of agreement. On modern embedding models, genuinely unrelated sentences often sit around 0.3-0.5, and merely topically-related ones — same domain, different question — routinely land at 0.80 and above. So 0.80 isn't the top of the scale; it's the murky middle where 'same topic' and 'same question' are no longer distinguishable.
The mechanism behind false hits. Embeddings encode topic and gist, which means the tokens that flip an answer without changing the topic move the vector very little. 'Cancel my order' and 'cancel my account' share the verb and the possessive and the support-request frame; their embeddings are close, comfortably past 0.80. 'Is this safe' and 'is this not safe' differ by one token and stay near-identical. A 0.80 gate waves all of these through.
The correction isn't a magic number — it's that the threshold must be chosen from data, by sweeping labeled pairs and reading the false-hit rate, and for an entity-heavy support domain that calibrated number is much higher than 0.80, or the intent is excluded entirely.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Intent tier | Caching policy | Why |
|---|---|---|
| Static FAQ (hours, policies) | Aggressive, calibrated threshold | Low harm if wrong, high repeat rate |
| Personalized status | Cache with per-user key + TTL | Correct but must not leak across users |
| High-stakes (billing, deletion) | Exclude from cache, serve live | False hit causes real, irreversible harm |
Real products, models, and research that use this idea.
- GPTCache exposes a configurable similarity threshold, and 0.80-class defaults are commonly too loose for entity-heavy support intents.
- Production support bots exclude billing, refunds, and account-deletion intents from the semantic cache and serve them live.
What an interviewer would ask next. Try answering before peeking at the approach.
QBumping the threshold to 0.95 reduces false hits — why isn't that the whole fix?
A higher threshold trades away recall and still can't separate every negation or entity swap; high-stakes intents need exclusion, not just a tighter number, because even a tiny false-hit rate is unacceptable there.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the semantic cache as one global config — a single similarity threshold over all intents — instead of a per-intent risk decision that excludes high-stakes requests.
60 second bullets to scan on the way to the call.
Why a cosine similarity of 0.80 counts as a loose threshold
Why near-miss queries clear 0.80 despite needing different answers
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.