Why does a cross-encoder reranker pay off even though it is slower than a bi-encoder retriever?

Flashcard·Easy·4.0 · 0·~30s·Asked atMckinsey Pinecone Sourcegraph

Attempt it

The bi-encoder retriever runs once against the whole corpus to fetch a top 50 or top 100 candidate set; it has to be fast. The cross-encoder reranker only runs over that small candidate set (tens, not millions), so its per-pair slowness is multiplied by a tiny number. In return it gives much better relevance ranking because it actually attends to the query and chunk together rather than comparing pre-computed embeddings. The pattern is cheap and shallow for first-stage recall, expensive and deep for second-stage precision, and the second stage is small enough that the absolute latency stays acceptable.

Your answer

The cross-encoder is slow per pair, but it does not run on the whole corpus. That is the key. The bi-encoder retriever runs first. It compares the query against pre-computed chunk embeddings via cosine similarity through an approximate nearest-neighbour index. That is fast, milliseconds even on a corpus of millions of chunks. The output is a wide candidate set, typically top 30 to top 100. The cross-encoder reranker runs second, but only over that small candidate set. For each (query, chunk) pair in the candidate set, it does one forward pass through the model where attention can compare query tokens to chunk tokens directly. Per pair, it is much slower than a cosine, tens of milliseconds versus microseconds. But the multiplier is small. Tens of candidates times tens of milliseconds each is still under a second of total reranker latency. The payoff is precision. The cross-encoder produces a relevance score that is sharper than cosine similarity because it sees the query and chunk jointly. The top-3 after reranking is much more likely to actually be the best 3 chunks than the top-3 from the bi-encoder alone. The pattern is cheap and shallow for first-stage recall (bi-encoder over the whole corpus), expensive and deep for second-stage precision (cross-encoder over the candidate set). The two stages do different jobs and each uses the model architecture that fits its job.

The reason a slow cross-encoder still pays off is that it never runs over the full corpus, it only runs over the small candidate set the bi-encoder already narrowed to. A bi-encoder embeds query and chunks independently. Chunk embeddings are precomputed once and stored in an approximate nearest-neighbour index (Faiss, Pinecone, Qdrant). At query time the system embeds the query once and finds the nearest k1 chunks by cosine, a few milliseconds end to end, regardless of corpus size. That speed is what makes first-stage retrieval feasible at scale. A cross-encoder takes a (query, chunk) pair as a single concatenated input and runs attention over both halves. The output is one relevance score that reflects token-level matching between query and chunk. Per pair, it is one full forward pass through the model, tens of milliseconds, even on hosted reranker APIs like Cohere Rerank 3.5 or BGE Reranker v2. The arithmetic that makes the pipeline work: corpus of one million chunks, bi-encoder retrieves top 30 in 5 ms, cross-encoder reranks the 30 in 20 ms. Total around 25 ms. If you tried to use the cross-encoder over the full corpus, it would be one million forward passes per query, minutes per query, completely infeasible. The payoff is precision. Bi-encoder cosine correlates with relevance in aggregate but mis-ranks at the top of the list. Cross-encoder relevance is much sharper because attention sees query and chunk together. The two-stage shape, wide and shallow first, narrow and deep second, is how production RAG gets both good recall and good precision without paying minutes of latency per query.

TL;DR

Cross-encoder slowness is multiplied by tens of candidates, not millions of corpus chunks, so total reranker latency stays acceptable while precision climbs sharply over bi encoder only ranking.

Memory aid

Easy to grasp

Picture sifting through a mountain of resumes. A super-careful interviewer who can spend twenty minutes on each one is too slow to read all five thousand. So a quick first pass narrows the pile to thirty. Then the careful interviewer spends twenty minutes each on those thirty and picks the actual best five. The careful interviewer is slow per resume but only sees thirty resumes total. That is the bi-encoder plus cross-encoder pattern in retrieval: cheap and broad first, expensive and careful second.

Key concepts

Context engineering Retrieval-augmented generation Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Cross-encoder rerankers look prohibitively slow when measured on per-pair latency. Tens of milliseconds per pair is glacial compared to the microseconds of a bi-encoder cosine. The reason the pattern still wins is that the cross-encoder is multiplied against a small number, the candidate set, not the corpus.

This card walks through the architectural reason the two models cost what they cost, the arithmetic that makes the pipeline work, and the precision gain that makes the cost worthwhile.

What each model architecture caches

A bi-encoder runs query and chunks through the model independently. Each output is an embedding vector. Because the chunk side does not depend on the query, every chunk in the corpus can be embedded once at ingest and stored in an approximate nearest neighbour index. At query time the system embeds the query (one forward pass) and runs an ANN lookup over the precomputed chunk index. The query-time cost is independent of corpus size, milliseconds whether the corpus has ten thousand chunks or ten million.

A cross-encoder runs query and chunk through the model jointly. The input is a single concatenated sequence [CLS] query [SEP] chunk [SEP] and the output is a single relevance score. Attention can compare every query token against every chunk token, which is what makes the score sharper. The catch: nothing about this can be precomputed. Every (query, chunk) pair is a fresh forward pass.

The structural consequence

Bi-encoders are corpus-side cacheable; cross-encoders are not. That asymmetry is the reason the two architectures live at different stages of the pipeline.

The arithmetic that makes it work

Why bi-encoder cosine mis-ranks at the top

When to skip the reranker

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import cohere, time

co = cohere.Client()

def retrieve_with_rerank(query: str, k1: int = 30, k2: int = 5):
    t0 = time.time()
    candidates = vector_db.search(query, top_k=k1)   # bi-encoder, ~5 ms over 1M chunks
    t1 = time.time()
    docs = [c.text for c in candidates]
    reranked = co.rerank(
        model="rerank-3.5",
        query=query,
        documents=docs,
        top_n=k2,
    )                                                  # cross-encoder, ~30 ms over 30 candidates
    t2 = time.time()
    print(f"retrieve {1000*(t1-t0):.1f} ms, rerank {1000*(t2-t1):.1f} ms")
    return [docs[r.index] for r in reranked.results]

Real products, models, and research that use this idea.

Cohere Rerank 3.5 is the dominant hosted cross-encoder in 2026, typically reranking 50-100 candidates in tens of milliseconds.
BGE Reranker v2 from BAAI is the leading open-weight cross-encoder, self-hostable for cost-sensitive workloads.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does k1 affect total reranker latency?

Roughly linear, doubling k1 doubles the reranker's forward-pass count. Production stacks set k1 large enough for recall but small enough that the reranker fits the latency budget.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Comparing the cross-encoder to the bi-encoder on per-pair latency. The right comparison is total latency including the candidate set size, where the cross-encoder over 30 candidates costs less than running it over a million.

60 second bullets to scan on the way to the call.

Explain why bi-encoder chunk embeddings can be precomputed and cross-encoder pairs cannot
State the typical k1 and k2 values for a 2026 production stack

Primary sources. Browse if you want the original framing.