Define recall@10 for a vector index, and spell out exactly what 'ground truth' means in that definition.

Short answer·Medium·4.0 · 0·~3 min·Asked atPinecone·Relevant atMicrosoft

Attempt it

Give a precise definition of recall@10 as used to evaluate a vector index, and explain what 'ground truth' refers to in that metric.

Free · 2 AI evals / day

Vector indexes are approximate. They trade exactness for speed. The question 'how approximate is it?' is answered by recall@K. **Recall@10 is a number between 0 and 1** that tells you what fraction of the true 10 nearest neighbors your ANN index actually returned. Pick a query, ask your index for its top 10. Separately, compute the true top 10 by exact brute-force search over the same corpus. Count how many of the index's 10 are in the true 10, divide by 10. Average that fraction over a held-out set of queries. **The 'ground truth' part matters.** The reference set has to be exact. The only way to get exact nearest neighbors in vector space is to compute the distance from the query to every vector in the corpus and sort. That is what Flat (brute-force) search does. Flat is too slow for production but is the unique honest ground-truth method for recall measurement. The practical recipe: run Flat once over your corpus on a held-out query set, save the ground-truth top-10 results to disk, then evaluate every ANN configuration against that frozen ground truth. The frozen file is what lets you compare HNSW with `ef_search = 100` against HNSW with `ef_search = 200` on the same axis.

**Definition.** Recall@10 is `|ANN_top10(q) intersect True_top10(q)| / 10`, averaged over a held-out query set Q. ANN_top10 is the top 10 results returned by the index under test. True_top10 is the top 10 nearest neighbors computed by exact (Flat) search over the same corpus under the same distance metric. **Why Flat is the only valid ground truth.** Flat computes the distance from the query to every vector in the corpus and returns the K closest. That is mathematically the definition of K-nearest-neighbors. Every other ANN method (HNSW, IVF, IVF-PQ, ScaNN) is an approximation and cannot serve as ground truth without circularity. If you measure 'recall' of method A against method B and both are approximate, you are measuring their agreement, not their correctness. **Three rules that get violated in practice.** First, the metric must match. Recall@10 under cosine cannot be compared to recall@10 under Euclidean even on the same corpus, because the top-10 nearest neighbors are different sets. Pick a metric, then compute ground truth under that metric, then evaluate ANN under that metric. Second, the corpus must match exactly. If you build the ANN index on corpus C and compute ground truth on corpus C' (because someone added documents in the meantime), the ground-truth file is wrong. Freeze a corpus snapshot before evaluating. Third, the query set must be held out. Queries should not overlap with documents in the corpus, and they should not have been used to train the embedding model. Otherwise the embeddings have memorized the answer and recall is inflated. **The standard public benchmarks** (SIFT1M, GIST1M, Deep1B, MS MARCO, BEIR subsets) ship as a corpus, a query set, and a precomputed ground-truth file specifically so everyone reports recall against the same reference. When a vendor reports a recall number without naming the dataset and metric, the number is uninterpretable.

Quick reference

TL;DR

Recall@10 is `|ANN_top10 intersect Exact_top10| / 10`, averaged over a held-out query set. Ground truth is the exact top-10 from Flat brute-force search over the same corpus under the same metric.

Memory aid

Easy to grasp

Imagine a librarian who claims they can find the 10 books most similar to yours in a giant library by walking only a few aisles. To check whether they really did the job well, you need a slow, honest comparison: someone who actually checks every single book in the library, takes their time, and writes down the true list of the 10 closest matches. Recall@10 is the score 'out of those 10 true closest matches, how many did the speedy librarian also find?' If they found 9 of the 10, recall is 0.9. The slow honest checker is the only fair judge, and they have to use the same definition of 'similar' as the speedy librarian.

Key concepts

Retrieval evaluation

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: precise recall@K formula + Flat as ground truth and why + metric/corpus/query-set discipline + recall as a curve over tuning knobs + reading vendor benchmark numbers honestly.

Side-by-side

What you compute	What it tells you	Valid?
\|ANN_K intersect Flat_K\| / K	Recall against true nearest neighbors	Yes, this is the definition
\|ANN_K(A) intersect ANN_K(B)\| / K	Agreement between two ANN methods	Not recall; meaningless as a quality metric
Mean reciprocal rank against Flat top-1	Ranking quality for the single best result	Useful, but different metric (MRR, not recall@K)
nDCG against labeled judgments	Relevance-graded retrieval quality	Useful for RAG end to end eval, not vector recall

Real-world examples

Real products, models, and research that use this idea.

ANN-Benchmarks (ann-benchmarks.com) is the open source evaluation harness that publishes recall-vs-QPS curves for every major ANN library; the ground-truth files ship with the dataset.
Faiss includes IndexFlatL2 and IndexFlatIP specifically so engineers can compute the reference top-K for their own corpora before measuring ANN recall.
The SIFT1M benchmark (1M 128-dim SIFT descriptors with 10k queries) is the canonical ANN testbed. Its ground-truth file is a published artifact reused across two decades of papers.
Pinecone's internal eval pipeline runs Flat over a frozen corpus snapshot weekly and uses that as the recall reference for every customer-facing benchmark number.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you compute ground truth at a billion vectors when Flat would take days?

QWhy is recall@10 the most common K in practice, and when would you measure recall@100 instead?

QHow does recall change as you tune ef_search in HNSW?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Using one ANN method's results as ground truth for evaluating another ANN method. You end up measuring agreement between two approximations, not recall against truth.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.