Semantic Cache
Also known as: Embedding cache
Cache LLM responses by embedding similarity, not exact string match; paraphrases hit the cache.
A cache keyed by the embedding of the prompt rather than its exact string. A new prompt whose embedding is close enough to a cached entry returns the cached response, cutting cost and latency for paraphrased queries.
In practice
Big cost lever for support bots and FAQs. Interviews probe the similarity threshold trade-off (false hits vs cache miss rate).
How it compares
Semantic cache compares whole prompts by meaning; prompt caching reuses common prompt prefixes byte-exactly.
Related topics
Questions that mention this term
- Order the cache layers an LLM request should check before paying for tokens
- Which levers cut RAG serving cost without gutting answer quality?
- Your production RAG costs $1M/month. The CFO wants this cut in half with a max 1 point faithfulness regression. What highest leverage cost optimizations do you deploy, in priority order?
- Where can caching cut cost in a RAG pipeline, and what does a semantic answer cache risk?
- A teammate randomized system prompts per user and expects the prompt cache to still help, what's wrong?
- Find the flaw: this semantic answer cache started serving answers to the wrong questions
Related terms
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
BM25
Classical TF-IDF-style lexical scoring: a surprisingly strong baseline, still the first stage of many hybrid RAG pipelines.
Chunking
Cutting documents into retrievable pieces before embedding them.
Context Engineering
Deciding what makes it into the model's context window (prompt + RAG + history + memory), and why.