Semantic Cache
Also known as: Embedding cache
Cache LLM responses by embedding similarity, not exact string match; paraphrases hit the cache.
A cache keyed by the embedding of the prompt rather than its exact string. A new prompt whose embedding is close enough to a cached entry returns the cached response, cutting cost and latency for paraphrased queries.
In practice
Big cost lever for support bots and FAQs. Interviews probe the similarity threshold trade-off (false hits vs cache miss rate).
How it compares
Semantic cache compares whole prompts by meaning; prompt caching reuses common prompt prefixes byte-exactly.
Related topics
Related terms
Retrieval-Augmented Generation (RAG)
Fetch relevant docs at inference time and stuff them into the prompt so the model can answer from real data.
Chunking
Cutting documents into retrievable pieces before embedding them.
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Context Window
The max number of tokens a model can attend to at once.
Hallucination
When a model confidently makes up something that isn't true.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.