Production recall dropped overnight with no deploy. Order the investigation from cheapest check to most invasive.
- 1Verify the alarm itself: rerun the recall eval and check whether the ground truth set or eval corpus snapshot went stale
- 2Rebuild or compact the index in staging from source data and compare recall to production to isolate index corruption or accumulated degradation
- 3Audit config drift: did a managed service or library auto upgrade silently change default ef_search, nprobe, or quantization settings
- 4Inspect index health stats: tombstone ratio, segment counts, and any large bulk delete or ingest jobs that ran overnight
- 5Probe the embedding provider: re-embed a fixed sentence set and diff against stored vectors to detect a silent model version change behind the same API name
Verify the metric, then read-only inspect stats and configs, then probe the embedding contract, and only then pay for a staging rebuild.
Imagine a smoke alarm going off in the middle of the night. The first thing a careful adult does is check the alarm itself. Maybe the battery is low and chirping, not actually detecting smoke. Next they walk around with their eyes open: is there visible smoke, is the stove on, did anything obvious change? Only after those quick checks do they call the fire department and start ripping out drywall. Production debugging is the same. The cheapest checks come first because they often catch the real cause and they never make things worse. Tearing apart the building is last because it is slow, risky, and only worth doing once everything cheap has been ruled out.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Incident triage in retrieval systems is a Bayes problem dressed up as a runbook. Every check has a cost (engineer time, production risk, infrastructure spend) and a hit rate (the probability it catches the actual cause). The optimal order is by hit rate divided by cost, descending. The cheap checks come first not because they are intellectually simpler but because they catch a large fraction of incidents in minutes and never make production worse. Expensive checks come last because they are slow, risky, and pay off only when nothing else explains the regression.
The specific ordering for an overnight recall drop with no deploy is metric, stats, configs, embedding contract, rebuild. That order encodes a strong prior: when nothing in your codebase changed, the thing that changed is almost always something silent. Silent things in vector search land in roughly four buckets: a stale or broken measurement, an internal index state that aged, a config flipped by someone or something other than you, or a provider that updated a model behind a stable name. The rebuild path is reserved for the case where none of those bear out.
This walkthrough lays out each step's mechanics, what evidence makes you escalate to the next step, and why the order is robust even when the actual cause turns out to be something exotic.
Step 1: verify the alarm
The single most common cause of an overnight recall regression with no deploy is that something about the measurement changed, not the system. Treat the alarm as a hypothesis and try to refute it before acting on it.
Rerun the recall eval against the same ground-truth snapshot from before the regression. If the number recovers, the eval pipeline introduced the false alarm. Check whether the truth IDs still point at the same content; a nightly job that re-embedded the corpus or rewrote document IDs can leave the truth set pointing at the wrong docs. Confirm the eval code itself has not been auto-deployed (some teams ship eval code on a different cadence than the index).
This is the cheapest check by far. It is one eval run on existing infrastructure, no production touch, no inference spend beyond the probes themselves. The hit rate is high enough that skipping this step is the most common preventable mistake in retrieval incidents.
If the eval recovers, you are done. Document the cause as a measurement drift and add a guardrail that flags ground-truth snapshot age the next time a metric moves.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Pinecone serverless reshards data plane segments on its own schedule; teams have caught recall dips traced to mid-reshard ef_search defaults.
- Voyage AI voyage-3 and OpenAI text-embedding-3-large have both had stable API names while internal versions changed, exactly the case the canonical-sentence probe catches.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you make steps 1-4 parallelizable to compress incident time?
Run them concurrently in a runbook script. Step 1 is fully isolated (eval rerun); step 2 is read-only stats; step 3 is config diff; step 4 is a small inference call. None block the others. The serial story is for explanation; the execution is parallel.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Jumping straight to a staging rebuild before checking whether the alarm itself is lying or whether a config auto-upgrade or silent embedding model change explains the regression.
60 second bullets to scan on the way to the call.
Why metric verification comes before any infrastructure inspection
What ground-truth drift looks like and how to detect it
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.