How would you detect that your production RAG vector index has gone stale, before users notice degraded answers?

Short answer·Medium·4.0 · 0·~3 min·Relevant atDatabricks Perplexity

Attempt it

You operate a production RAG system over a knowledge corpus that updates daily (new docs added, old docs revised or removed). Describe three concrete signals you'd instrument to detect that your vector index has gone stale, before user complaints about wrong or outdated answers reach support.

Free · 2 AI evals / day

A **stale index** means the vector store no longer reflects the live corpus: new documents aren't indexed yet, deleted ones are still searchable, or revised ones serve old text. The LLM happily answers from whatever it retrieves, so the bad answer looks confident. You want to catch this before users do. The trick is to instrument three signals at different speeds. The fastest is a freshness check: stamp every chunk with when its source last changed and when you indexed it, then alert when that gap grows past your refresh cadence. For a daily update corpus, anything more than ~48 hours behind is a red flag. The second signal is a golden-set eval. Keep a fixed list of test questions where you already know the ideal chunks, and re-run retrieval nightly. If recall drops, retrieval has degraded for queries you trust. The third signal watches real traffic: the distribution of similarity scores, the refusal rate, the share of low-confidence retrievals. When scores sag or refusals spike, the index is matching queries worse than before. Used together, the first warns earliest and the third catches what the others miss.

I'd instrument three signals, ordered from leading to lagging, because each catches a different staleness mode. **Freshness assertions (leading)**. Every chunk carries a source modified timestamp and an indexed at timestamp. I emit the lag between them per source and alert when p99 exceeds the refresh cadence. For a daily corpus, page when any source is more than ~48 hours behind. This is the most direct signal. The timestamps literally say the index is behind, before retrieval quality has visibly moved. **Golden-set retrieval eval (mid-lag)**. I maintain a fixed set of (query, ideal chunk) pairs and re-run retrieval nightly, computing context recall and the rank of the ideal chunk against a baseline. When recall drops past the noise floor, retrieval has degraded for known good queries. This catches subtler cases where chunks exist but no longer rank highly. **Query-stream metrics (lagging)**. On live traffic I watch the top-k cosine score distribution, the low-confidence retrieval rate, and the refusal rate. A drop in mean top-1 score or a spike in refusals flags degradation on real queries the golden set never anticipated. The three compose cleanly: freshness answers "is the data current", golden-set answers "does retrieval still work for known queries", and stream metrics answer "is real traffic degrading". Stream-only monitoring alone lags by hours to days, which is exactly the window where users notice first.

Quick reference

TL;DR

Detect a stale RAG index with three signals: per-chunk freshness lag (leading), nightly golden-set retrieval recall (mid-lag), and query-stream score/refusal metrics (lagging). Together they catch every staleness mode.

Memory aid

Easy to grasp

Imagine a library where the catalog is supposed to match the shelves. Over time new books arrive, old ones get edited or thrown out, but the catalog doesn't always keep up. You want to catch this before a reader complains that the book they found was the wrong edition. So you watch three things. First, you check the dates: does every catalog card say it was updated recently, or are some weeks behind? Second, you keep a list of test questions you already know the right shelf for, and each night you ask the catalog those questions to see if it still points to the right place. Third, you watch real readers: if more of them suddenly leave empty handed or grab books that barely match, something is off. The first check warns you earliest; the third catches surprises the first two missed.

Key concepts

RAG system design Vector databases LLM evaluation Embeddings

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name three signals, map each to a staleness mode, order them leading to lagging with detection latency, then give a concrete threshold or cadence for each and explain why one alone is insufficient.

Side-by-side

Signal	Staleness mode caught	Latency (leading vs lagging)	Typical cadence / threshold
Freshness assertions (source vs indexed timestamp)	Data staleness: new/revised/deleted docs not propagated	Leading	Continuous; alert p99 lag > refresh cadence (~48h for daily)
Golden-set retrieval eval (context recall, ideal chunk rank)	Retrieval-quality staleness: chunks present but rank dropped	Mid-lag	Nightly; alert recall drop past noise floor vs baseline
Query-stream metrics (top-k score, refusal, low-confidence rate)	Real-traffic staleness: degradation on unseen query patterns	Lagging	Continuous; alert on distribution shift / refusal spike

Real-world examples

Real products, models, and research that use this idea.

Glean's enterprise search re-crawls connected SaaS sources on a cadence and tracks per-connector freshness lag, so stale documents surface as an ingestion alert rather than a bad answer.
The RAGAS-on-CI pattern lets teams run a fixed golden eval set nightly in 2026, computing context recall and faithfulness to catch retrieval regressions before deploy.
Arize Phoenix and LangSmith dashboards instrument live RAG traces (retrieval score distributions, low-confidence rates) so query-stream drift is visible without waiting for support tickets.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QFreshness is green but golden-set recall just dropped 8%. What are the likely causes and how do you triage?

QHow do you build a golden set that doesn't itself go stale as the corpus evolves?

QRaw cosine scores aren't calibrated across embedding models or corpora. How do you make the stream signal trustworthy?

QYour corpus has hard deletes (GDPR right to erasure). How does that change your staleness detection?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Naming only one signal, usually waiting for query-stream metrics or user complaints to dip. That is the slowest, most lagging detector and misses staleness for days.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.