Flashcard: is a 0.85 cosine similarity score from your retriever 'high confidence'?

No, a **cosine similarity** of 0.85 is not a calibrated confidence score. It is a relative ranking number, not an absolute probability. Here is what it actually means. The retriever turned your query and every chunk into vectors, then measured the angle between them. A 0.85 says this chunk points more in the query's direction than the chunks scoring 0.84 or 0.83. It ranks them. It does not say there is an 85 percent chance the chunk is relevant. The number depends entirely on the query and the embedding model. For one query the best chunk might score 0.85; for another, equally good query the best might score 0.55. The scale slides around. This breaks two tempting shortcuts. First, a fixed threshold like 'keep everything above 0.7' will silently return nothing on some queries and noise on others. Second, you cannot compare a 0.85 from one embedding model against a 0.85 from another; they live in different vector spaces. Use the score to rank, then evaluate relevance separately.

No. A 0.85 cosine score is a **relative** ranking signal, not an absolute, calibrated probability of relevance. Cosine measures the angle between the query vector and the chunk vector. A 0.85 only tells you this chunk is more aligned with the query than the ones scoring lower, in this retriever's space, for this query. It is not '85 percent likely to be relevant', and the model was never trained to make the raw value mean that. Two consequences matter in interviews. First, thresholding on an absolute value is a classic bug. The score distribution shifts per query. So 'keep chunks above 0.7' can drop every chunk on a hard query and admit junk on an easy one. Second, comparing cosine scores across different embedding models or retrievers is meaningless: different spaces, different score distributions. What to do instead: use the score to rank within a query, take top-k, and judge actual quality with a separate signal. A cross-encoder reranker gives a far better relevance estimate. If you need a hard accept/reject gate, calibrate a threshold against a labeled eval set per model, or threshold on the reranker's score, not the raw cosine.