A vendor's benchmark chart plots QPS against recall@10. What should you look for, and what claim should make you skeptical?

MCQ·Hard·4.0 · 0·~1 min·Asked atPinecone·Relevant atNVIDIA

Attempt it

Vector database vendors love QPS charts. Big numbers, dramatic curves, your eye is drawn to the peak. The problem is that QPS and recall are linked through the index's tuning knobs. Cranking down `ef_search` in HNSW or `nprobe` in IVF lets the index answer faster but with less accuracy. Every point on the curve is a (recall, QPS) pair, and the vendor gets to pick which point they put in big font on the slide. **The right way to read the curve.** First, decide what recall your product actually needs. RAG with a strong re-ranker can tolerate recall 0.85. Security retrieval probably needs 0.99+. Find that recall on the x-axis of the chart. Read the QPS off the curve at that point. That is the number that matters to you. **What makes a claim trustworthy.** Five pieces of metadata: the dataset (SIFT1M? Your own corpus?), the embedding model (text-embedding-3-large at 3072 dim is very different from all-MiniLM-L6 at 384 dim), the dimensionality, the distance metric (cosine, dot, Euclidean), and the source of ground truth (Flat, please, not 'another ANN method'). A vendor missing any of these is hiding something. Why the other options are wrong. There is no industry-standard recall level; vendors absolutely cherry-pick. Peak QPS at recall 0.7 is useless if your product needs 0.95. And recall and QPS are not independent; they are two values of the same tuning sweep.

**The chart's two axes are connected by the index's tuning knobs.** Sliding `ef_search`, `nprobe`, or `pq_m` traces out the curve. Picking one point as 'the' headline number is choosing which tuning the vendor wants you to see, which is rarely the tuning you would actually deploy. **Right read in three steps.** Step 1. Determine your recall SLO. This is a product question, not a vector-database question. RAG-with-reranker can tolerate 0.85. RAG-without-reranker needs 0.92+. Security retrieval and compliance retrieval need 0.99+. Write your number down before looking at the chart. Step 2. Locate your recall on the x-axis. Read QPS off the curve at that recall. Do that for every vendor on the chart. That is the comparison. Step 3. Sanity-check the metadata. The five things to look for: dataset, embedding model, embedding dimension, distance metric, source of ground truth. Each one changes the result by a lot. SIFT1M (128-dim) and Deep1B (96-dim) benchmark very differently from a modern 1024-dim or 3072-dim text embedding. Cosine and Euclidean produce different rankings except on normalized vectors. Ground truth must be Flat to be sound. **Why headline QPS is meaningless without recall.** Any ANN library can hit absurdly high QPS at recall 0.5. The information content of '10,000 QPS' is zero without the recall. The information content of '10,000 QPS at recall 0.95 on SIFT1M with the publishedly-trained 128-dim SIFT descriptors under L2, ground truth from Flat' is high. **Why the other options are wrong.** There is no industry-standard recall level; vendors choose where on the curve to highlight, often deliberately picking a low-recall regime where their tuning happens to win. Peak QPS at low recall is useless if your product needs high recall. And recall is not independent of QPS; they trace a curve, and the tuning knob picks the operating point. **ANN-Benchmarks is the trustworthy alternative.** The open-source ann-benchmarks project runs every major ANN

Quick reference

TL;DR

Read the curve at YOUR recall SLO, not at the vendor's chosen highlight point. Be skeptical of any QPS number missing the recall, the dataset, the embedding model and dim, and the metric.

Memory aid

Easy to grasp

Imagine two car dealers each post a flyer claiming their car is fastest. One says '180 mph!' on a closed track. The other says '60 mph in 4 seconds!' in city traffic. Which is faster? You cannot compare them because they tested in different conditions and reported different things. Vector database benchmarks are the same. The number 'we hit 10,000 queries per second' is only useful if you know how accurate those answers were and on what data the test ran. The right way to compare two vendors is to draw both their curves on the same axes (accuracy on one axis, speed on the other) and read the speed at the accuracy your product needs.

Key concepts

ANN benchmarking

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: how tuning knobs trace the recall vs QPS curve + reading the curve at your SLO + five metadata items + p99 latency + ANN-Benchmarks as neutral ground + how vendor charts mislead.

Side-by-side

What the vendor shows	What to actually compute	Why
Peak QPS at unspecified recall	QPS at your product's recall SLO	Peak is usually at recall the vendor cherry-picked
Average latency	p99 latency	User facing systems are bottlenecked by the tail, not the mean
Benchmark on SIFT1M only	Benchmark on the same embedding model you will deploy	Data geometry changes results materially
Recall vs (other ANN)	Recall vs Flat	Only Flat is exact; non-Flat ground truth is circular

Real-world examples

Real products, models, and research that use this idea.

ANN-Benchmarks publishes recall vs QPS curves for every major library on standardized datasets. It is the de facto neutral comparison source in 2026 and most reputable vendors mirror their numbers there.
Big-ANN-Benchmarks (NeurIPS competition track) runs at 100M to 1B scale and grades systems on recall, throughput, AND build time / memory, which the marketing charts almost never show.
Pinecone, Qdrant, Weaviate, and Milvus all publish their own marketing benchmarks. When two vendors disagree, ANN-Benchmarks is the tie-breaker.
The Big-ANN streaming track tests indexes under continuous upserts plus queries, which surfaces failure modes (tombstone accumulation, recall drift) that static benchmarks miss entirely.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat does it mean for a recall-QPS curve to dominate another?

QHow would you design your own ANN benchmark for an internal vendor selection?

QWhy does build time belong on the comparison even though it does not appear on recall-QPS curves?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Comparing two vendors' headline QPS numbers without checking that both were measured at the same recall, on the same dataset, with the same embedding model and dimension.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.