Give a precise definition of recall@10 as used to evaluate a vector index, and explain what 'ground truth' refers to in that metric.
Free · 2 AI evals / day
Vector indexes are approximate. They trade exactness for speed. The question 'how approximate is it?' is answered by recall@K.
**Recall@10 is a number between 0 and 1** that tells you what fraction of the true 10 nearest neighbors your ANN index actually returned. Pick a query, ask your index for its top 10. Separately, compute the true top 10 by exact brute-force search over the same corpus. Count how many of the index's 10 are in the true 10, divide by 10. Average that fraction over a held-out set of queries.
**The 'ground truth' part matters.** The reference set has to be exact. The only way to get exact nearest neighbors in vector space is to compute the distance from the query to every vector in the corpus and sort. That is what Flat (brute-force) search does. Flat is too slow for production but is the unique honest ground-truth method for recall measurement.
The practical recipe: run Flat once over your corpus on a held-out query set, save the ground-truth top-10 results to disk, then evaluate every ANN configuration against that frozen ground truth. The frozen file is what lets you compare HNSW with `ef_search = 100` against HNSW with `ef_search = 200` on the same axis.
**Definition.** Recall@10 is `|ANN_top10(q) intersect True_top10(q)| / 10`, averaged over a held-out query set Q. ANN_top10 is the top 10 results returned by the index under test. True_top10 is the top 10 nearest neighbors computed by exact (Flat) search over the same corpus under the same distance metric.
**Why Flat is the only valid ground truth.** Flat computes the distance from the query to every vector in the corpus and returns the K closest. That is mathematically the definition of K-nearest-neighbors. Every other ANN method (HNSW, IVF, IVF-PQ, ScaNN) is an approximation and cannot serve as ground truth without circularity. If you measure 'recall' of method A against method B and both are approximate, you are measuring their agreement, not their correctness.
**Three rules that get violated in practice.**
First, the metric must match. Recall@10 under cosine cannot be compared to recall@10 under Euclidean even on the same corpus, because the top-10 nearest neighbors are different sets. Pick a metric, then compute ground truth under that metric, then evaluate ANN under that metric.
Second, the corpus must match exactly. If you build the ANN index on corpus C and compute ground truth on corpus C' (because someone added documents in the meantime), the ground-truth file is wrong. Freeze a corpus snapshot before evaluating.
Third, the query set must be held out. Queries should not overlap with documents in the corpus, and they should not have been used to train the embedding model. Otherwise the embeddings have memorized the answer and recall is inflated.
**The standard public benchmarks** (SIFT1M, GIST1M, Deep1B, MS MARCO, BEIR subsets) ship as a corpus, a query set, and a precomputed ground-truth file specifically so everyone reports recall against the same reference. When a vendor reports a recall number without naming the dataset and metric, the number is uninterpretable.
Quick reference
TL;DR
Recall@10 is `|ANN_top10 intersect Exact_top10| / 10`, averaged over a held-out query set. Ground truth is the exact top-10 from Flat brute-force search over the same corpus under the same metric.
Imagine a librarian who claims they can find the 10 books most similar to yours in a giant library by walking only a few aisles. To check whether they really did the job well, you need a slow, honest comparison: someone who actually checks every single book in the library, takes their time, and writes down the true list of the 10 closest matches. Recall@10 is the score 'out of those 10 true closest matches, how many did the speedy librarian also find?' If they found 9 of the 10, recall is 0.9. The slow honest checker is the only fair judge, and they have to use the same definition of 'similar' as the speedy librarian.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Comparing two indexes built under different metrics The True_top10 sets are different objects; recall numbers are not on the same axis and cannot be averaged or ranked against each other.
Reporting recall on a live, drifting production corpus Ground truth is implicitly tied to a snapshot. The recall number drifts as the corpus changes, not because the index got worse, and you cannot tell the two apart.
Using a downstream RAG quality metric (faithfulness, helpfulness) as a substitute for recall Those metrics confound retrieval quality with generation quality; you need a clean retrieval-side number to isolate the index choice.
Quick scan · 2–4 min
Everything important, quickly.
4 min: precise recall@K formula + Flat as ground truth and why + metric/corpus/query-set discipline + recall as a curve over tuning knobs + reading vendor benchmark numbers honestly.
Side-by-side
What you compute
What it tells you
Valid?
|ANN_K intersect Flat_K| / K
Recall against true nearest neighbors
Yes, this is the definition
|ANN_K(A) intersect ANN_K(B)| / K
Agreement between two ANN methods
Not recall; meaningless as a quality metric
Mean reciprocal rank against Flat top-1
Ranking quality for the single best result
Useful, but different metric (MRR, not recall@K)
nDCG against labeled judgments
Relevance-graded retrieval quality
Useful for RAG end to end eval, not vector recall
Real-world examples
Real products, models, and research that use this idea.
1ANN-Benchmarks (ann-benchmarks.com) is the open source evaluation harness that publishes recall-vs-QPS curves for every major ANN library; the ground-truth files ship with the dataset.
2Faiss includes IndexFlatL2 and IndexFlatIP specifically so engineers can compute the reference top-K for their own corpora before measuring ANN recall.
3The SIFT1M benchmark (1M 128-dim SIFT descriptors with 10k queries) is the canonical ANN testbed. Its ground-truth file is a published artifact reused across two decades of papers.
4Pinecone's internal eval pipeline runs Flat over a frozen corpus snapshot weekly and uses that as the recall reference for every customer-facing benchmark number.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you compute ground truth at a billion vectors when Flat would take days?
Sign in or go Pro to see the approach for this follow-up.
QWhy is recall@10 the most common K in practice, and when would you measure recall@100 instead?
Sign in or go Pro to see the approach for this follow-up.
QHow does recall change as you tune ef_search in HNSW?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Using one ANN method's results as ground truth for evaluating another ANN method. You end up measuring agreement between two approximations, not recall against truth.