Design a CI gate that blocks deploys when vector index recall regresses.

Short answer·Medium·4.0 · 0·~3 min·Asked atNeo4j Persistent Weaviate

Attempt it

After an incident where an innocent looking config change silently dropped retrieval recall for a week, your team wants a CI gate: any change to index config, embedding pipeline, or database version must prove recall before it ships. Design the harness: what data it uses, what it measures, what thresholds gate the merge, and how you keep it fast enough for CI.

Free · 2 AI evals / day

Recall regressions are silent. Latency dashboards do not catch them. The fix is a **CI gate that proves recall before any change ships**. **The data.** A pinned query set (a few hundred to a few thousand representative queries) plus a frozen corpus snapshot. **Ground truth** is the true nearest neighbors for each query, computed once by exact Flat search and stored as a versioned artifact. Regenerate ground truth only when the corpus snapshot or the embedding model version changes. **The harness.** In CI, build the candidate index over the snapshot (or a stratified sample), replay the query set, compute recall@k for the production k values (e.g. recall@10, recall@100), and report latency percentiles too. **The gate.** Fail the merge if recall@10 drops more than a small tolerance (typically 1 point absolute) versus the recorded baseline. Allow an explicit override path with sign-off for intentional tradeoffs. Update the baseline on merge. **Speed.** Cache embeddings (never re-embed in CI), keep the sample small enough to build in minutes, parallelize the query replay, and run the full-size variant nightly. **Triggers.** Gate on index config changes, embedding model bumps, and database or library version upgrades. The last one bites teams because vendor upgrades sometimes change default parameters.

**Four pieces: data, harness, gate, speed.** Plus the operational discipline of which changes the gate covers. - **Data.** Pinned evaluation set of 500 to 2,000 queries representative of production. Frozen corpus snapshot stored in object storage. Ground truth (true top-k nearest neighbors per query) computed once via exact Flat search and stored as a versioned artifact alongside the snapshot. Ground truth is regenerated only when the snapshot or embedding model version changes. - **Harness.** In CI, fetch the snapshot (or a stratified sample large enough to exercise the index type), build the candidate index using the proposed configuration, replay the query set against it, compute recall@k for the production k values (typically recall@10, recall@100), and capture latency p50/p95/p99 as secondary signals. - **Gate.** Compare against a recorded baseline. Fail the merge if recall@10 drops by more than the agreed tolerance (1 point absolute is a common starting point). Provide an explicit override label (`recall-override-approved`) with required reviewer sign-off for intentional tradeoffs (e.g. enabling PQ for memory savings). On merge, update the stored baseline. - **Speed.** Cache the embedded vectors so CI never re-embeds. Use a stratified sample (e.g. 1M of 200M vectors) for per-commit runs, building in 2 to 5 minutes. Parallelize query replay across CPU cores. Run the full-size variant nightly to catch effects that only show up at scale. - **Coverage.** The gate triggers on index config changes (HNSW M, ef_construction), embedding pipeline changes (model version, normalization), and database or library version bumps. The last one is the most often-forgotten and the most likely culprit in silent regressions, because vendor upgrades change defaults.

TL;DR

Pin a query set and corpus snapshot, compute ground truth with exact Flat search once, build the candidate index in CI, measure recall@k against a recorded baseline with tolerance, and gate every config, embedding,

Memory aid

Easy to grasp

Picture a kitchen where the chef must taste every new recipe against a reference dish before serving it. The reference dish is made once and refrigerated. Every time a cook changes the recipe (new ingredient, new oven, new appliance brand), they prepare the new version, taste it side by side with the reference, and only serve if the new version is at least as good. The full reference batch is huge so for quick tastings the cook uses a small sample. The full batch goes to the head chef once a day. If anyone tries to ship a recipe that scored lower without explicit signoff, the door is locked.

Key concepts

Vector Databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Recall regressions are insidious because they pass every other check. Tests pass. Latency stays flat. Error rates do not budge. The only signal is that users get slightly worse results, and that signal only shows up when somebody runs a manual evaluation or a customer complains.

The canonical incident pattern is: someone bumps the database library version. The new version ships with a different default for ef_construction or ef_search. The index gets rebuilt with the new defaults at the next snapshot. Recall drops 5 points. Nobody notices for a week.

A CI gate solves this by making recall a regression-protected metric. The rest of this section walks through the four pieces (data, harness, gate, speed), then covers the operational discipline that makes the gate actually work in practice.

The artifacts: snapshot, query set, ground truth

Three versioned artifacts live in object storage and form the backbone of the gate.

Corpus snapshot. A frozen copy of the production corpus at a specific point in time. Stored as raw documents plus their embeddings (cached so CI never re-embeds). For corpora under 10M vectors, the full snapshot is usable. For larger corpora, a stratified sample preserves the distribution at a fraction of the size.

Query set. 500 to 2,000 queries representative of production traffic. Stratified to include head queries, tail queries, hard cases, and queries from each intent type (factoid, semantic, navigational). Composition matters more than size; a 500-query set that covers the regimes catches more regressions than a 5,000-query set that is all head traffic.

Ground truth. For each query, the true top-k nearest neighbors over the snapshot, computed by exact Flat search. Stored as a versioned artifact alongside the snapshot. Ground truth is the expensive thing to compute (linear in corpus size per query), and the whole point of storing it is to amortize that cost across many CI runs.

Regeneration rule: ground truth is regenerated only when the corpus snapshot or the embedding model version changes. Configuration changes do not require regeneration because they do not affect what the true nearest neighbors are; they only affect how well the candidate index approximates them.

Version all three artifacts together. A snapshot and a ground truth that drift out of sync silently corrupts the gate.

The harness and the gate

Coverage breadth: what changes trigger the gate

Speed engineering and operational discipline

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LangSmith offers an evaluation harness for RAG that follows this pattern: pinned eval set, ground-truth top-k, recall and precision metrics
Pinecone publishes recall benchmark templates that teams adapt into CI gates around their index configs

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you set the recall tolerance in a principled way?

Run the gate 10 times against the same configuration with different random seeds. Measure the standard deviation of recall. Set tolerance to roughly 2 sigma above the no-op variance. Below that, you get false positives; above, you miss small regressions.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Building a recall harness that runs only on index config changes, missing the silent regressions from embedding model bumps or vendor version upgrades.

60 second bullets to scan on the way to the call.

What artifacts need to be versioned for a recall gate to work
How ground truth is computed and when it must be regenerated

Primary sources. Browse if you want the original framing.