Design the cache key for an embedding cache that survives vendor model upgrades

Attempt it

You're building an embedding cache to avoid re-embedding the same content twice. Design the cache key. Explain why a naive content-hash key is wrong, and what additional fields you need to include for correctness across model versions and fine-tunes.

Free · 2 AI evals / day

Quick reference

TL;DR

Cache key must be (content_hash, model_id, model_version, normalization_state) at minimum. Content alone collides across models and silent version bumps leak stale vectors.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a library where every book gets photocopied, but a different machine sits in each room. Each machine produces slightly different copies. If you label your photocopies only by the book title, you cannot tell which machine made which copy. The day someone asks for the photocopy of Moby Dick, you might hand them a version that does not match the one currently in use. A saved lookup table for these text fingerprints works the same way. The same paragraph turns into different numbers depending on which fingerprinting machine produced it, so the label needs to record both the text and the machine. Otherwise, the day someone upgrades the machines, every saved answer becomes silently wrong, with no error, just bad search results.

Key concepts

Embeddings Vector databases RAG

Quick scan · 2–4 min

Everything important, quickly.

9 minutes: cross-model collision, the full key shape, silent-bump detection, Matryoshka and normalization wrinkles, and invalidation policy.

Code example

python

import hashlib
from dataclasses import dataclass

@dataclass(frozen=True)
class EmbeddingCacheKey:
    content_hash: str       # sha256 over normalized input text
    model_id: str           # 'text-embedding-3-large'
    model_version: str      # snapshot date or model-card hash
    embedding_dim: int      # 1536, 3072, etc. (Matryoshka-aware)
    normalized: bool        # True if unit-norm, False if raw
    instruction_hash: str   # hash of instruction prefix; '' if none

    def to_str(self) -> str:
        parts = (
            self.content_hash,
            self.model_id,
            self.model_version,
            str(self.embedding_dim),
            str(int(self.normalized)),
            self.instruction_hash,
        )
        return "|".join(parts)

def make_key(text: str, model_id: str, model_version: str,
             dim: int, normalized: bool, instruction: str = "") -> EmbeddingCacheKey:
    content = hashlib.sha256(text.strip().encode("utf-8")).hexdigest()
    instr   = hashlib.sha256(instruction.encode("utf-8")).hexdigest() if instruction else ""
    return EmbeddingCacheKey(content, model_id, model_version, dim, normalized, instr)

Real-world examples

Real products, models, and research that use this idea.

LangChain's CacheBackedEmbeddings wrapper namespaces cache entries by model id, sidestepping cross-model collision.
LlamaIndex's IngestionCache uses a fingerprint that includes model parameters for the same reason.
Pinecone's recommended migration pattern uses a parallel index keyed by model version to avoid mixed-vintage retrieval during cutover.
OpenAI's text-embedding-ada-002 behavior changes in 2023 are the canonical example of why model_version belongs in the key, even when the name is stable.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you detect a vendor silent model bump if the model_id name does not change?

Sign in or go Pro to see the approach for this follow-up.

QWould you cache vectors keyed by a hash of the raw text or the normalized text?

Sign in or go Pro to see the approach for this follow-up.

QHow do you avoid a thundering-herd re-embed when the cache invalidates on a model swap?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.