Why does a cross-encoder reranker pay off even though it is slower than a bi-encoder retriever?

Flashcard·Easy·4.0 · 0·~30s·Asked atCohere Databricks OpenAI

Attempt it

The bi-encoder retriever runs once against the whole corpus to fetch a top 50 or top 100 candidate set; it has to be fast. The cross-encoder reranker only runs over that small candidate set (tens, not millions), so its per-pair slowness is multiplied by a tiny number. In return it gives much better relevance ranking because it actually attends to the query and chunk together rather than comparing pre-computed embeddings. The pattern is cheap and shallow for first-stage recall, expensive and deep for second-stage precision, and the second stage is small enough that the absolute latency stays acceptable.

Your answer

The cross-encoder is slow per pair, but it does not run on the whole corpus. That is the key. The bi-encoder retriever runs first. It compares the query against pre-computed chunk embeddings via cosine similarity through an approximate nearest-neighbour index. That is fast, milliseconds even on a corpus of millions of chunks. The output is a wide candidate set, typically top 30 to top 100. The cross-encoder reranker runs second, but only over that small candidate set. For each (query, chunk) pair in the candidate set, it does one forward pass through the model where attention can compare query tokens to chunk tokens directly. Per pair, it is much slower than a cosine, tens of milliseconds versus microseconds. But the multiplier is small. Tens of candidates times tens of milliseconds each is still under a second of total reranker latency. The payoff is precision. The cross-encoder produces a relevance score that is sharper than cosine similarity because it sees the query and chunk jointly. The top-3 after reranking is much more likely to actually be the best 3 chunks than the top-3 from the bi-encoder alone. The pattern is cheap and shallow for first-stage recall (bi-encoder over the whole corpus), expensive and deep for second-stage precision (cross-encoder over the candidate set). The two stages do different jobs and each uses the model architecture that fits its job.

The reason a slow cross-encoder still pays off is that it never runs over the full corpus, it only runs over the small candidate set the bi-encoder already narrowed to. A bi-encoder embeds query and chunks independently. Chunk embeddings are precomputed once and stored in an approximate nearest-neighbour index (Faiss, Pinecone, Qdrant). At query time the system embeds the query once and finds the nearest k1 chunks by cosine, a few milliseconds end to end, regardless of corpus size. That speed is what makes first-stage retrieval feasible at scale. A cross-encoder takes a (query, chunk) pair as a single concatenated input and runs attention over both halves. The output is one relevance score that reflects token-level matching between query and chunk. Per pair, it is one full forward pass through the model, tens of milliseconds, even on hosted reranker APIs like Cohere Rerank 3.5 or BGE Reranker v2. The arithmetic that makes the pipeline work: corpus of one million chunks, bi-encoder retrieves top 30 in 5 ms, cross-encoder reranks the 30 in 20 ms. Total around 25 ms. If you tried to use the cross-encoder over the full corpus, it would be one million forward passes per query, minutes per query, completely infeasible. The payoff is precision. Bi-encoder cosine correlates with relevance in aggregate but mis-ranks at the top of the list. Cross-encoder relevance is much sharper because attention sees query and chunk together. The two-stage shape, wide and shallow first, narrow and deep second, is how production RAG gets both good recall and good precision without paying minutes of latency per query.

Quick reference

TL;DR

Cross-encoder slowness is multiplied by tens of candidates, not millions of corpus chunks, so total reranker latency stays acceptable while precision climbs sharply over bi encoder only ranking.

Memory aid

Easy to grasp

Picture sifting through a mountain of resumes. A super-careful interviewer who can spend twenty minutes on each one is too slow to read all five thousand. So a quick first pass narrows the pile to thirty. Then the careful interviewer spends twenty minutes each on those thirty and picks the actual best five. The careful interviewer is slow per resume but only sees thirty resumes total. That is the bi-encoder plus cross-encoder pattern in retrieval: cheap and broad first, expensive and careful second.

Key concepts

Context engineering Retrieval-augmented generation Embeddings

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 minutes: bi-encoder caching versus cross-encoder no-caching, the arithmetic over the candidate set size, why bi-encoder cosine mis-ranks at the top, name two production rerankers.

Code example

python

import cohere, time

co = cohere.Client()

def retrieve_with_rerank(query: str, k1: int = 30, k2: int = 5):
    t0 = time.time()
    candidates = vector_db.search(query, top_k=k1)   # bi-encoder, ~5 ms over 1M chunks
    t1 = time.time()
    docs = [c.text for c in candidates]
    reranked = co.rerank(
        model="rerank-3.5",
        query=query,
        documents=docs,
        top_n=k2,
    )                                                  # cross-encoder, ~30 ms over 30 candidates
    t2 = time.time()
    print(f"retrieve {1000*(t1-t0):.1f} ms, rerank {1000*(t2-t1):.1f} ms")
    return [docs[r.index] for r in reranked.results]

Real-world examples

Real products, models, and research that use this idea.

Cohere Rerank 3.5 is the dominant hosted cross-encoder in 2026, typically reranking 50-100 candidates in tens of milliseconds.
BGE Reranker v2 from BAAI is the leading open-weight cross-encoder, self-hostable for cost-sensitive workloads.
Voyage Rerank-2 specializes in technical and code domains and is commonly paired with Voyage embedding models.
Jina Reranker v2 ships an open-weight cross-encoder with long-context support up to 8K tokens.
Pinecone, Weaviate, and Qdrant all expose hosted reranker integration as a single config flag in 2026.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does k1 affect total reranker latency?

QWhat happens if you push k1 to 200 or 500?

QCan you replace the cross-encoder with a smaller LLM rerank?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Comparing the cross-encoder to the bi-encoder on per-pair latency. The right comparison is total latency including the candidate set size, where the cross-encoder over 30 candidates costs less than running it over a million.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.