Spot the error in this hybrid search scoring code that combines BM25 and cosine scores.

Spot the error·Medium·4.0 · 0·~2 min·Asked atAlibaba Amd Observe Ai

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Hybrid search uses two scoring systems for the same documents: BM25 for keyword matching and cosine similarity for semantic matching. To combine them you need one number per document. The buggy code adds the raw scores together. The problem is the two numbers live on different scales. BM25 has no upper bound; on a real corpus you see scores from 5 to 40. Cosine similarity is bounded between -1 and 1. When you add a BM25 score of 18 to a cosine score of 0.3, the cosine contribution is one part in sixty. It might as well be zero. The consequence: every ranking decision is made by BM25 alone. The dense vector arm is in the code, runs at query time, costs CPU, and contributes nothing. That is silent failure. Two fixes work. **Reciprocal rank fusion** converts both lists to ranks, then sums `1 / (60 + rank)` across the lists. Ranks are scale-free. **Per-arm normalization** rescales each arm's scores to the same range (min-max or z-score within the candidate set) before adding. Reciprocal rank fusion is the production default because it has no parameters to tune per-corpus and produces consistent results across domains.

**The bug is scale incompatibility.** BM25 produces unbounded positive scores, commonly 5-40 on production corpora; cosine similarity is bounded in [-1, 1] and on a healthy retrieval is typically 0.2-0.8. Summing them lets BM25 dominate by a factor of 30-60x. A document with BM25 18.2 and cosine 0.31 has final = 18.51. A document with BM25 17.9 and cosine 0.95 has final = 18.85. The first document is barely ahead on BM25 but the second is dramatically ahead on cosine. The buggy code picks the first. **Why this is a silent failure.** Output looks plausible. Top-10 results contain real documents. The bug is detectable only by comparing against an evaluation set or by inspecting cases where the two arms disagree. **The two fixes.** - **Reciprocal rank fusion.** Convert each list to ranks (position 1 to 100). Score each doc as `sum(1 / (60 + rank_in_list))` over the lists where it appears. The constant 60 is the Cormack 2009 recommendation. Scale-free, no per-corpus tuning, robust default. - **Per-arm normalization plus weighted sum.** Rescale each arm's scores within the candidate set (min-max to [0, 1] or z-score), then take a weighted sum with a tunable alpha. Slightly more powerful when you have evaluation data to tune alpha, but requires that tuning to be redone per corpus. Production engines (OpenSearch, Weaviate, Qdrant, Pinecone) all ship RRF as the default. The fix in the code is one line: replace `doc.bm25_score + doc.cosine_score` with the RRF sum over ranks.

TL;DR

BM25 scores can reach 40 while cosine sits in [-1, 1]. Adding them lets BM25 silently dominate. Fix it with reciprocal rank fusion or with explicit per-arm normalization.

Memory aid

Easy to grasp

Pretend you are judging a baking contest where one judge scores from 1 to 10 and another scores from 1 to 1000. If you add the two scores, the second judge wins every time, even when their cake was clearly worse. The first judge is invisible. Hybrid search has the same problem. The keyword judge (BM25) hands out scores like 5 to 40. The meaning judge (cosine) hands out scores between -1 and 1. Add them and BM25 always wins, even when cosine had a strong opinion. The fix is to convert both judges' opinions into ranks, where first place is first place regardless of the absolute score. That is what reciprocal rank fusion does, and it is why every modern hybrid engine ships RRF by default.

Key concepts

Vector Databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Adding raw BM25 and cosine scores is the single most common hybrid-search bug in production. It survives code review, looks fine in spot checks, and silently degrades retrieval quality for natural-language queries while preserving it for exact-match queries, which means product analytics that mix both query types fail to surface the bug.

This walkthrough covers the mechanism (why the two scoring systems live on incompatible scales), the canonical fix (reciprocal rank fusion), the operational variants (alpha-weighted fusion, per-arm normalization), and how to detect the bug in production without a labelled evaluation set.

Mental model: BM25 hands out points like a tournament with no ceiling; cosine grades on a [-1, 1] curve. Adding scores from different rulebooks lets the larger ruler always win. The fix is to compare ranks, which are scale-free.

Why the scales are incompatible

BM25 is the sum over query terms of IDF(term) * tf_saturation(term, doc). The IDF component for a term that appears in 10 of a 100M-document corpus is log(100M / 10) = 16. With three rare terms in the query, BM25 contributions stack to 40-plus. There is no cap on the score; rare-term queries on a large corpus routinely produce scores in the 20-50 range.

Cosine similarity between two unit vectors is bounded in [-1, 1] by the Cauchy-Schwarz inequality:

\text{cos}(u, v) = \frac{u \cdot v}{\lVert u \rVert \, \lVert v \rVert}

On a healthy retrieval workload the top-100 cosine scores typically span 0.2 to 0.8. Sometimes higher for near-duplicates, lower for noisy queries.

The two numbers cannot be added as if they were on the same axis. A BM25 of 18 and a cosine of 0.3 sum to 18.3. A BM25 of 17 and a cosine of 0.9 sum to 17.9. The first wins, even though the second arm's signal was overwhelmingly stronger. The dense arm's contribution is statistical noise relative to the BM25 contribution; it might as well not run.

The failure is not numerical instability or any subtle thing. It is that the linear sum of incommensurate scores is meaningless as a ranking. The fix has to either rescale the scores into a shared range or replace the score with something that is already shared, like ranks.

Reciprocal rank fusion as the canonical fix

Per-arm normalization plus weighted sum

Detecting the bug in production

Why score normalization matters more than the fusion function

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

# Reciprocal Rank Fusion: merge BM25 and dense rankings without score normalization.
# rank starts at 1; k=60 is the Cormack 2009 default and is robust across corpora.
def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranked in rankings:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: -kv[1])

# Usage: dense and sparse arms each return ranked doc_ids.
fused = rrf([dense_results, bm25_results], k=60)
# WRONG: scores = [a + b for a, b in zip(bm25_scores, cosine_scores)]
# BM25 ~ [0, 40], cosine ~ [-1, 1]; the sum is effectively BM25 alone.

Real products, models, and research that use this idea.

OpenSearch hybrid query ships RRF as a normalization processor, the default fusion method for sparse plus dense pipelines.
Weaviate hybrid search uses RRF with a tunable alpha for arm weighting, exposed directly in the query API.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you detect this bug in production without a labelled evaluation set?

Use disagreement queries as a proxy. Sample queries where BM25 and dense top results overlap by less than 30 percent. On those queries, compare the raw-sum top-3 against an A/B group running RRF top-3. User click-through and dwell time on the RRF arm reveals the lift even without explicit labels.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing the bug is small because BM25 and cosine 'should' agree. They often disagree, and on those queries the raw-sum approach silently picks BM25's opinion every time.

60 second bullets to scan on the way to the call.

Why BM25 and cosine produce scores on incompatible scales
How the scale mismatch lets BM25 dominate ranking decisions silently

Primary sources. Browse if you want the original framing.