Flashcard: is a 0.85 cosine similarity score from your retriever 'high confidence'?

Attempt it

No. Cosine similarity is RELATIVE, not absolute or calibrated. 0.85 means 'this chunk is closer to the query than 0.84, 0.83, ... in THIS retriever's vector space for THIS query'. It does NOT mean '85% probability of being relevant'. Two consequences: (1) thresholding on absolute cosine values (e.g. 'keep chunks > 0.7') is a classic bug; the threshold that worked for one query may produce empty results for another. (2) Comparing cosine scores ACROSS retrievers or across different embedding models is meaningless; they live in different vector spaces with different score distributions.

Your answer

Quick reference

TL;DR

A 0.85 cosine score is relative, not calibrated. It ranks this chunk against others for this query; it is not an 85% probability of relevance, and is meaningless across models.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a race where everyone gets a number for how close they finished to the winner. A 0.85 just means this runner came in ahead of the ones who got 0.84 or 0.83 in that one race. It does not mean they ran well, or that they had an 85 percent chance of winning. Now run the race on a different track, with a different judge who measures distances differently. The exact same runner gets a totally different number, because the scale itself changed. So the number is great for putting runners in order within one race. But you can never read it as a grade like 'this runner is 85 percent good,' and you can never compare a number from one race to a number from another. To know who actually performed well, you need a separate, fair judge.

Key concepts

Embeddings Vector databases Retrieval-Augmented Generation LLM evaluation

Quick scan · 2–4 min

Everything important, quickly.

3 min: explain cosine is relative not calibrated, the two failure modes (global threshold, cross-model comparison), and the reranker plus eval-set fix.

Code example

python

# BAD: absolute threshold treated as calibrated confidence
hits = index.query(qvec, top_k=20)
good = [h for h in hits if h.score > 0.7]  # breaks per query / per model

# BETTER: rank with cosine, then judge relevance with a reranker
hits = index.query(qvec, top_k=50)            # cosine = candidate generation
ranked = reranker.rerank(query, [h.text for h in hits])
good = [r for r in ranked if r.relevance_score > 0.5]  # calibrated signal

Side-by-side

Property	Raw cosine score	Reranker / calibrated score
Meaning	Relative rank in this space	Estimate of actual relevance
Comparable across models	No, different vector spaces	Closer, trained for relevance
Safe to threshold globally	No, distribution shifts per query	More defensible with eval set
Cost	Cheap, one dot product	Higher, cross-encoder per pair

Real-world examples

Real products, models, and research that use this idea.

Pinecone and Weaviate return raw similarity scores that teams wrongly threshold globally; vendor docs in 2026 explicitly warn the values are not calibrated probabilities.
Cohere Rerank 3.5 is sold precisely as the calibrated relevance layer to put after dense retrieval instead of trusting raw cosine.
RAGAS context-precision lets teams measure retrieval quality against ground truth rather than guessing a cosine cutoff.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build a defensible accept/reject gate for retrieved chunks if cosine isn't calibrated?

Sign in or go Pro to see the approach for this follow-up.

QWhy do most embedding models produce cosine scores clustered in a narrow high range rather than spread across 0 to 1?

Sign in or go Pro to see the approach for this follow-up.

QIf you must use a per-query threshold, how would you make it adaptive?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.