Find the flaw: this semantic answer cache started serving answers to the wrong questions

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

The bug is the cosine threshold of 0.75 — too loose for a semantic answer cache. Distinct questions sharing topic words collide above it, so it serves wrong answers. A tight bar near 0.95 admits only true paraphrases.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a help desk that keeps a folder of past answers. When a new question comes in, the clerk checks if it 'feels similar' to one already answered and, if so, hands over that old answer instead of doing the work again. The trouble is the clerk's bar for 'similar' is way too low. 'How do I reset my password' and 'how do I change my password' feel similar — they share most of the same words — but they want different answers. With a loose bar, the clerk confidently hands over the wrong one. The fix is to make the clerk much stricter, only reusing an old answer when the new question is nearly word for word the same.

Key concepts

Retrieval-Augmented Generation Embeddings

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: identify the loose 0.75 threshold, explain why caching needs precision unlike retrieval, show how topical neighbors collide, and cover calibration plus intent gating and invalidation as the rest of a correct cache.

Real-world examples

Real products, models, and research that use this idea.

GPTCache and similar semantic-cache libraries expose a similarity threshold; setting it too low causes cross-question collisions in production.
Support assistants caching 'reset password' answers and serving them to 'change password' queries because both exceed a loose cosine bar.
Teams calibrating the cache threshold against a labeled set of true-paraphrase vs near-miss query pairs instead of hardcoding a round number.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you actually choose the cache threshold rather than guessing 0.95?

Sign in or go Pro to see the approach for this follow-up.

QBeyond the threshold, what else stops a semantic cache from serving stale or wrong answers?

Sign in or go Pro to see the approach for this follow-up.

QWhy is cosine similarity a poor calibrated measure of semantic equivalence here?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating cosine similarity as a calibrated quality score and picking a low threshold like 0.75 the way you would for retrieval recall — caching needs precision, so the bar has to be near-identical, not just topically close.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

GPTCache: A Library for Creating Semantic Cache for LLM Queries