Vector database vendors love QPS charts. Big numbers, dramatic curves, your eye is drawn to the peak. The problem is that QPS and recall are linked through the index's tuning knobs. Cranking down `ef_search` in HNSW or `nprobe` in IVF lets the index answer faster but with less accuracy. Every point on the curve is a (recall, QPS) pair, and the vendor gets to pick which point they put in big font on the slide.
**The right way to read the curve.** First, decide what recall your product actually needs. RAG with a strong re-ranker can tolerate recall 0.85. Security retrieval probably needs 0.99+. Find that recall on the x-axis of the chart. Read the QPS off the curve at that point. That is the number that matters to you.
**What makes a claim trustworthy.** Five pieces of metadata: the dataset (SIFT1M? Your own corpus?), the embedding model (text-embedding-3-large at 3072 dim is very different from all-MiniLM-L6 at 384 dim), the dimensionality, the distance metric (cosine, dot, Euclidean), and the source of ground truth (Flat, please, not 'another ANN method'). A vendor missing any of these is hiding something.
Why the other options are wrong. There is no industry-standard recall level; vendors absolutely cherry-pick. Peak QPS at recall 0.7 is useless if your product needs 0.95. And recall and QPS are not independent; they are two values of the same tuning sweep.
**The chart's two axes are connected by the index's tuning knobs.** Sliding `ef_search`, `nprobe`, or `pq_m` traces out the curve. Picking one point as 'the' headline number is choosing which tuning the vendor wants you to see, which is rarely the tuning you would actually deploy.
**Right read in three steps.**
Step 1. Determine your recall SLO. This is a product question, not a vector-database question. RAG-with-reranker can tolerate 0.85. RAG-without-reranker needs 0.92+. Security retrieval and compliance retrieval need 0.99+. Write your number down before looking at the chart.
Step 2. Locate your recall on the x-axis. Read QPS off the curve at that recall. Do that for every vendor on the chart. That is the comparison.
Step 3. Sanity-check the metadata. The five things to look for: dataset, embedding model, embedding dimension, distance metric, source of ground truth. Each one changes the result by a lot. SIFT1M (128-dim) and Deep1B (96-dim) benchmark very differently from a modern 1024-dim or 3072-dim text embedding. Cosine and Euclidean produce different rankings except on normalized vectors. Ground truth must be Flat to be sound.
**Why headline QPS is meaningless without recall.** Any ANN library can hit absurdly high QPS at recall 0.5. The information content of '10,000 QPS' is zero without the recall. The information content of '10,000 QPS at recall 0.95 on SIFT1M with the publishedly-trained 128-dim SIFT descriptors under L2, ground truth from Flat' is high.
**Why the other options are wrong.** There is no industry-standard recall level; vendors choose where on the curve to highlight, often deliberately picking a low-recall regime where their tuning happens to win. Peak QPS at low recall is useless if your product needs high recall. And recall is not independent of QPS; they trace a curve, and the tuning knob picks the operating point.
**ANN-Benchmarks is the trustworthy alternative.** The open-source ann-benchmarks project runs every major ANN
Quick reference
TL;DR
Read the curve at YOUR recall SLO, not at the vendor's chosen highlight point. Be skeptical of any QPS number missing the recall, the dataset, the embedding model and dim, and the metric.
Imagine two car dealers each post a flyer claiming their car is fastest. One says '180 mph!' on a closed track. The other says '60 mph in 4 seconds!' in city traffic. Which is faster? You cannot compare them because they tested in different conditions and reported different things. Vector database benchmarks are the same. The number 'we hit 10,000 queries per second' is only useful if you know how accurate those answers were and on what data the test ran. The right way to compare two vendors is to draw both their curves on the same axes (accuracy on one axis, speed on the other) and read the speed at the accuracy your product needs.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Picking a vendor based on a single QPS number in a marketing slide The number is one cherry-picked point on a curve and tells you almost nothing about how the system will behave at your operating point.
Benchmarking on SIFT1M when you will deploy 3072-dim text embeddings Data geometry changes the recall vs QPS curve substantially; benchmark on the actual embedding model you will use.
Comparing systems on average latency only User-facing systems are constrained by p99; an average that looks fine can hide a long tail that breaks SLOs.
Quick scan · 2–4 min
Everything important, quickly.
4 min: how tuning knobs trace the recall vs QPS curve + reading the curve at your SLO + five metadata items + p99 latency + ANN-Benchmarks as neutral ground + how vendor charts mislead.
Side-by-side
What the vendor shows
What to actually compute
Why
Peak QPS at unspecified recall
QPS at your product's recall SLO
Peak is usually at recall the vendor cherry-picked
Average latency
p99 latency
User facing systems are bottlenecked by the tail, not the mean
Benchmark on SIFT1M only
Benchmark on the same embedding model you will deploy
Data geometry changes results materially
Recall vs (other ANN)
Recall vs Flat
Only Flat is exact; non-Flat ground truth is circular
Real-world examples
Real products, models, and research that use this idea.
1ANN-Benchmarks publishes recall vs QPS curves for every major library on standardized datasets. It is the de facto neutral comparison source in 2026 and most reputable vendors mirror their numbers there.
2Big-ANN-Benchmarks (NeurIPS competition track) runs at 100M to 1B scale and grades systems on recall, throughput, AND build time / memory, which the marketing charts almost never show.
3Pinecone, Qdrant, Weaviate, and Milvus all publish their own marketing benchmarks. When two vendors disagree, ANN-Benchmarks is the tie-breaker.
4The Big-ANN streaming track tests indexes under continuous upserts plus queries, which surfaces failure modes (tombstone accumulation, recall drift) that static benchmarks miss entirely.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat does it mean for a recall-QPS curve to dominate another?
Sign in or go Pro to see the approach for this follow-up.
QHow would you design your own ANN benchmark for an internal vendor selection?
Sign in or go Pro to see the approach for this follow-up.
QWhy does build time belong on the comparison even though it does not appear on recall-QPS curves?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Comparing two vendors' headline QPS numbers without checking that both were measured at the same recall, on the same dataset, with the same embedding model and dimension.