Define recall@10 for a vector index, and spell out exactly what 'ground truth' means in that definition.

Short answer·Medium·4.0 · 0·~3 min·Asked atDroom Ola Pinecone·Relevant atMicrosoft NVIDIA

Attempt it

Give a precise definition of recall@10 as used to evaluate a vector index, and explain what 'ground truth' refers to in that metric.

Free · 2 AI evals / day

Vector indexes are approximate. They trade exactness for speed. The question 'how approximate is it?' is answered by recall@K. **Recall@10 is a number between 0 and 1** that tells you what fraction of the true 10 nearest neighbors your ANN index actually returned. Pick a query, ask your index for its top 10. Separately, compute the true top 10 by exact brute-force search over the same corpus. Count how many of the index's 10 are in the true 10, divide by 10. Average that fraction over a held-out set of queries. **The 'ground truth' part matters.** The reference set has to be exact. The only way to get exact nearest neighbors in vector space is to compute the distance from the query to every vector in the corpus and sort. That is what Flat (brute-force) search does. Flat is too slow for production but is the unique honest ground-truth method for recall measurement. The practical recipe: run Flat once over your corpus on a held-out query set, save the ground-truth top-10 results to disk, then evaluate every ANN configuration against that frozen ground truth. The frozen file is what lets you compare HNSW with `ef_search = 100` against HNSW with `ef_search = 200` on the same axis.

**Definition.** Recall@10 is `|ANN_top10(q) intersect True_top10(q)| / 10`, averaged over a held-out query set Q. ANN_top10 is the top 10 results returned by the index under test. True_top10 is the top 10 nearest neighbors computed by exact (Flat) search over the same corpus under the same distance metric. **Why Flat is the only valid ground truth.** Flat computes the distance from the query to every vector in the corpus and returns the K closest. That is mathematically the definition of K-nearest-neighbors. Every other ANN method (HNSW, IVF, IVF-PQ, ScaNN) is an approximation and cannot serve as ground truth without circularity. If you measure 'recall' of method A against method B and both are approximate, you are measuring their agreement, not their correctness. **Three rules that get violated in practice.** First, the metric must match. Recall@10 under cosine cannot be compared to recall@10 under Euclidean even on the same corpus, because the top-10 nearest neighbors are different sets. Pick a metric, then compute ground truth under that metric, then evaluate ANN under that metric. Second, the corpus must match exactly. If you build the ANN index on corpus C and compute ground truth on corpus C' (because someone added documents in the meantime), the ground-truth file is wrong. Freeze a corpus snapshot before evaluating. Third, the query set must be held out. Queries should not overlap with documents in the corpus, and they should not have been used to train the embedding model. Otherwise the embeddings have memorized the answer and recall is inflated. **The standard public benchmarks** (SIFT1M, GIST1M, Deep1B, MS MARCO, BEIR subsets) ship as a corpus, a query set, and a precomputed ground-truth file specifically so everyone reports recall against the same reference. When a vendor reports a recall number without naming the dataset and metric, the number is uninterpretable.

TL;DR

Recall@10 is `|ANN_top10 intersect Exact_top10| / 10`, averaged over a held-out query set. Ground truth is the exact top-10 from Flat brute-force search over the same corpus under the same metric.

Memory aid

Easy to grasp

Imagine a librarian who claims they can find the 10 books most similar to yours in a giant library by walking only a few aisles. To check whether they really did the job well, you need a slow, honest comparison: someone who actually checks every single book in the library, takes their time, and writes down the true list of the 10 closest matches. Recall@10 is the score 'out of those 10 true closest matches, how many did the speedy librarian also find?' If they found 9 of the 10, recall is 0.9. The slow honest checker is the only fair judge, and they have to use the same definition of 'similar' as the speedy librarian.

Key concepts

Retrieval evaluation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Recall@K looks like a simple metric until you try to compute it correctly. The formula is one line. The discipline around constructing the ground truth, holding the corpus fixed, matching the metric, holding out queries, is where most production evals go wrong. The question is asking for both halves: the formula and the ground-truth discipline.

The formula and what each term means

Recall@K is a set-overlap metric. For each query in a held-out query set Q, you have two sets of size K: what the ANN index returned, and what the true nearest neighbors actually are. The per query recall is the size of their intersection divided by K. The reported recall@K for the system is the mean of these per query values across Q.

It is bounded between 0 and 1. Recall = 1 means the ANN index returned exactly the true top-K (in any order; order is not graded by this metric). Recall = 0 means none of the true top-K appeared.

It is not the only retrieval metric. nDCG, MRR, and precision-at-K all exist and grade different things. Recall@K specifically asks 'what fraction of the true neighbors did you find', which is the right metric when the downstream system (a re-ranker, a cross-encoder, a human) will inspect the K candidates and is willing to ignore the others.

Why Flat is the unique valid ground-truth method

The three rules that get violated in production evals

Recall is a curve, not a point

Situations where this technique stops working.

2–4 min · Everything important, quickly.

What you compute	What it tells you	Valid?
\|ANN_K intersect Flat_K\| / K	Recall against true nearest neighbors	Yes, this is the definition
\|ANN_K(A) intersect ANN_K(B)\| / K	Agreement between two ANN methods	Not recall; meaningless as a quality metric
Mean reciprocal rank against Flat top-1	Ranking quality for the single best result	Useful, but different metric (MRR, not recall@K)
nDCG against labeled judgments	Relevance-graded retrieval quality	Useful for RAG end to end eval, not vector recall

Real products, models, and research that use this idea.

ANN-Benchmarks (ann-benchmarks.com) is the open source evaluation harness that publishes recall-vs-QPS curves for every major ANN library; the ground-truth files ship with the dataset.
Faiss includes IndexFlatL2 and IndexFlatIP specifically so engineers can compute the reference top-K for their own corpora before measuring ANN recall.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you compute ground truth at a billion vectors when Flat would take days?

Sub-sample to a representative slice (say 10M vectors) for ground truth, or use GPU-accelerated Flat (faiss-gpu) which is roughly 100x faster than CPU. Some teams accept partial ground truth on a few thousand queries rather than the full set.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using one ANN method's results as ground truth for evaluating another ANN method. You end up measuring agreement between two approximations, not recall against truth.

60 second bullets to scan on the way to the call.

Recall@K is intersection size over K, averaged over queries
Flat is the unique valid ground-truth method (exactness)

Primary sources. Browse if you want the original framing.