Define recall@k in the embedding retrieval context

Flashcard·Easy·4.0 · 0·~30s·Asked atLtimindtree Sigmoid

Attempt it

Given a labeled set of (query, relevant_doc_ids) pairs, recall@k for one query is the fraction of its relevant docs that appear in the top-k retrieved results. The metric averages this over queries. Many RAG evals use the hit form (hit rate@k): the fraction of queries where AT LEAST ONE relevant doc appears in the top-k. The two are equal when each query has one relevant doc. So with one relevant doc per query, recall@10 = 0.85 means 85% of queries had it in the top 10. It's the standard headline metric for embedding-retrieval quality but is incomplete: it ignores ordering within the top-k and confounds embedding quality with ANN index quality.

Your answer

**Recall@k** is the standard headline metric for retrieval systems. Given a labeled evaluation set where each query has known relevant documents, recall@k asks what share of those documents appear in the top-k retrieved results. Many RAG evals use the hit form: for what fraction of queries does at least one relevant document appear in the top-k? With one relevant document per query, the two match. Here's the hit calculation. You have 100 evaluation queries. For each, run retrieval, look at the top 10 results, and ask 'did at least one relevant doc appear?'. The fraction of yes answers is recall@10. So recall@10 = 0.85 means that in 85% of queries, at least one relevant document made it into the top 10. Recall@k directly measures whether the system surfaces relevant content within a budget of k results. But it has two big blind spots. First, it ignores ordering within the top-k: a relevant doc at rank 1 and one at rank 10 count the same. Second, on a production ANN (approximate nearest-neighbor) index, you measure the embedding and the index together, not the embedding alone. Metrics like NDCG@k and MRR capture ordering; using an exact (flat) index isolates embedding quality.

**Recall@k** is the share of a query's relevant documents that appear in the top-k retrieved results, averaged over a labeled evaluation set. Many RAG evals use the hit form: the proportion of queries where at least one relevant document appears in the top-k. The two match when each query has one relevant document. The hit calculation. For each (query, relevant_doc_ids) pair, run retrieval and score the query 'hit' if at least one relevant doc is in the top-k, 'miss' otherwise. Recall@k is the fraction of hits. For example: 100 queries, top-10 for each. 85 queries have at least one relevant doc in the top-10. Recall@10 = 0.85. For RAG, k is often the chunk budget you'll pass to the LLM (k=5 or k=10 are common). So recall@k directly tells you whether the LLM will see relevant context. Recall@k has two important limitations. First, recall@k ignores ordering within the top-k. A relevant doc at rank 1 and a relevant doc at rank 10 contribute equally. But the downstream LLM behavior differs: lost-in-the-middle effects mean rank-1 content has outsized impact on the answer. Use NDCG@k or MRR to capture rank sensitivity. Second, recall@k against an approximate ANN index (HNSW, IVF-PQ) confounds embedding quality with index quality. The result depends on both the embedding's true recall and the ANN index's recall against an exact baseline. To isolate embedding quality, run the eval against an exact (flat) index. To measure the ANN trade-off separately, compare ANN recall@k to flat recall@k.

TL;DR

Recall@k is the share of a query's relevant documents found in the top-k, averaged over queries. Many RAG evals use the hit form: the share of queries with at least one relevant document in the top-k.

Memory aid

Easy to grasp

Picture asking a librarian for any book about World War II and watching them grab the top 10 from the shelf. Recall@10 asks one question: did at least one actual WW2 book make it into those 10? If yes, the librarian got credit for the query. Recall@10 across many queries is the fraction of times the librarian got credit. (That is the hit form many RAG evals use. The textbook form asks what share of all the WW2 books made it into the 10.) The blind spot: the metric doesn't care whether the WW2 book was the first one handed over or buried at position 10. It only cares that it showed up somewhere in the pile.

Key concepts

Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Recall@k is the metric you'll encounter first in any retrieval evaluation and the metric you'll outgrow last. It's clean, easy to compute, and easy to compare across systems. It's also incomplete in ways that matter for production decisions.

Its two main blind spots are rank ignorance and the index confound, and companion metrics fill the gaps.

Definition and calculation

Recall@k starts with a labeled evaluation set. Each entry in the set is a pair: a query, and a list of document IDs that are known to be relevant to that query. The labels typically come from human annotation or search logs (click-through as a weak signal). They can also come from strong LLM graders run over a candidate pool.

For each query, run the retrieval system and take the top-k results. Compare the top-k document IDs to the labeled relevant IDs. In the textbook IR form, the query scores the fraction of its labeled relevant IDs found in the top-k. In the hit form that many RAG evals use, the query scores 1 (hit) if at least one labeled relevant ID is in the top-k, 0 (miss) otherwise. The two match when each query has one relevant ID.

Recall@k is the mean of these per query scores across the evaluation set. A hit-form recall@10 of 0.85 across 100 queries means 85 queries had at least one relevant doc in their top-10 retrieval.

The metric has clean operational meaning. For a RAG pipeline that passes k=10 chunks to the LLM, recall@10 directly answers 'in what fraction of queries will the LLM see relevant context?'. That framing is what makes recall@k the right starting metric for most retrieval evals.

Recall@k versus precision@k

Blind spot one: rank within the top-k

Blind spot two: embedding versus index confound

Ground truth incompleteness and false negatives

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

BEIR benchmarks all report NDCG@10 as the primary metric, but recall@k breakouts are also published for downstream analysis.
MTEB's retrieval category reports NDCG@10 in the headline; per-task recall@k is available in the detailed breakdown.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does measuring recall@k on an HNSW index confound embedding quality and index quality?

HNSW's recall against an exact baseline is < 1 and depends on parameters like ef_search, M, and the data distribution. If model A produces vectors that HNSW handles well and model B produces vectors that HNSW handles poorly, model A can score higher on production HNSW even if model B has better intrinsic embeddings. The way to separate is to measure both against an exact (flat) index.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing recall@k with precision@k. Recall@k asks 'what share of the relevant docs made it in?' (the hit form asks 'did at least one?'). Precision@k asks 'how many of the top-k are relevant?' They measure different things and can move in opposite directions.

60 second bullets to scan on the way to the call.

Recall@k as the share of a query's relevant docs in the top-k, and the hit form (at least one relevant doc in the top-k)
Calculation walk-through (query loop, hit/miss decision, mean)

Primary sources. Browse if you want the original framing.