Why is recall@k an incomplete metric for embedding-model quality?

**Recall@k** is the standard headline metric, but it has three blind spots that can mislead you about which embedding model is actually better. Blind spot one: it confounds embedding quality with ANN-index quality. Production retrieval uses approximate nearest-neighbor indexes like HNSW. These are approximate by design: they trade some recall against the exact baseline for big speed gains. When you measure recall@10 on an HNSW index, you're measuring `embedding_recall \times index_recall`. Model A might have better embeddings but worse HNSW behavior; Model B might have worse embeddings but better HNSW behavior. The combined recall@10 can rank them backwards. Fix: measure against an exact (flat) index when comparing embedding models. Blind spot two: it ignores rank within the top-k. A relevant doc at rank 1 and a relevant doc at rank 10 score the same on recall@10. But the downstream impact differs. For an LLM consumer, lost-in-the-middle effects mean rank-1 content has outsized weight on the answer. NDCG@k and MRR are rank-sensitive metrics that capture this. A mature eval reports recall alongside these. Blind spot three: it depends on the labels being complete, which they usually aren't. If a relevant document wasn't labeled as relevant (a false negative in the ground truth), the model gets penalized for retrieving it. This bites dense semantic retrievers hardest because they find related content that lexical-keyword annotation might have missed. Fix: sample a subset of 'misses' and have an LLM grader re-check whether they're actually relevant. The combined recommendation: report recall@k AND NDCG@10 against an exact index, on a labeled set with sampled false-negative auditing.

**Three blind spots** explain why recall@k alone is incomplete for embedding-model quality. First, recall@k confounds embedding quality with ANN-index quality. Production indexes (HNSW, IVF-PQ, ScaNN) are approximate; they recover some fraction of the exact top-k for speed gains. Measure recall@10 on HNSW and you're measuring `embedding_recall \times index_recall`. A model that produces HNSW-friendly geometry can score higher than a model with better intrinsic embeddings but worse index behavior. The conclusion can reverse entirely. Fix: evaluate on an exact (flat) index during model selection, then accept a tunable ANN penalty in production. Second, recall@k ignores ordering within the top-k. Rank 1 and rank 10 score the same. Downstream impact differs sharply. For LLM consumers, lost-in-the-middle effects mean rank-1 content has dramatically more weight on the answer than rank-10. For human consumers, position 1 has dramatically higher click-through. Fix: rank-sensitive metrics. NDCG@k (logarithmic discount), MRR (1/rank of first relevant), MAP. Mature evals report recall@k alongside NDCG@10. Third, recall@k depends on complete ground truth, which semantic-retrieval labels never are. False negatives (relevant docs not labeled relevant) silently penalize models that retrieve them. This bites dense embedders hardest because they surface semantically related content keyword-based annotation misses. A new model retrieving better semantic matches can score lower than an old one retrieving only lexical matches in the labels. Fix: sampled false-negative auditing. Sample misses, have an LLM grader (Claude Opus 4.7, GPT-5.5) re-check, adjust scores upward. The combined recommendation: report NDCG@10 AND recall@10 against an exact flat index, on a labeled set with sampled false-negative auditing. Treat recall@k as necessary but insufficient.