Define cross-encoder reranking and when it's used

Flashcard·Easy·4.0 · 0·~30s·Asked atCohere

Attempt it

A second-stage scoring step that runs a cross-encoder over the top-K candidates returned by a first-stage (embedding) retriever. The cross-encoder produces a more accurate relevance score per pair but is too slow to run over the full corpus. Typical numbers: bi-encoder ANN gets the top-100; cross-encoder rescores those 100; the top-10 by cross-encoder score is what the LLM consumer sees.

Your answer

**Cross-encoder reranking** is the second stage of a two-stage retrieval system. The first stage uses bi-encoder embeddings: each document in the corpus is embedded once at index time, and the query is embedded at query time. A vector database does an approximate nearest neighbor search to return the top-100 most similar documents. The cross-encoder then takes each of those 100 candidates and runs a fresh transformer forward pass over the pair `(query, document)` to produce a relevance score. Unlike the bi-encoder, the cross-encoder sees the query and the document together in the same context, so the attention mechanism can compare specific tokens across them. That cross-attention is what makes the score so much sharper. The catch is cost. A cross-encoder forward pass on a query-document pair is far slower than a vector lookup. You cannot run it over a million documents per query in a tight latency budget. But running it on 100 candidates is cheap enough to fit in 100-300 ms. The pipeline shape is: embedding retrieval gets the top-100 fast; cross-encoder rerank picks the top-10 precisely; the LLM consumer reads those 10.

**Cross-encoder reranking** is the precision stage of canonical two-stage retrieval. Stage one is a bi-encoder, where the query and each document are embedded separately and compared by cosine similarity in an ANN index. Stage two is the cross-encoder: a transformer that takes a `(query, document)` pair as a single concatenated input and outputs one relevance score per forward pass. The quality lift comes from cross-attention. A bi-encoder has to compress each side of the pair into a fixed-dim vector before comparing, so it cannot model fine-grained token interactions between query and document. A cross-encoder sees both inputs together and lets every query token attend to every document token. That extra interaction sharpens relevance dramatically; on BEIR a cross-encoder typically lifts NDCG@10 by 5-15 points over bi-encoder alone. The latency cost is the reason cross-encoders are not used in stage one. A cross-encoder forward pass is a full transformer inference per (query, doc) pair, so running it over a million-document corpus would take seconds per query. Running it over the 100 candidates surfaced by the bi-encoder takes 100-300 ms on a modern reranker like Cohere rerank-v3 or BGE-reranker-v2. The canonical 2026 production stack: bi-encoder embeddings (OpenAI text-embedding-3-large, Voyage v3) → ANN to top-100 → cross-encoder rerank (Cohere rerank, Jina rerank, BGE-reranker) to top-10 → LLM consumer.

Quick reference

TL;DR

A precision-focused second stage that rescores the first-stage's top-K with a cross-encoder, picking the sharpest top-10 for the LLM consumer.

Memory aid

Easy to grasp

Picture finding the best lawyer for a case. First you skim a hundred profiles fast, narrow down to the twenty that look promising, and only then do real interviews with those twenty. The skim is cheap and lets you cover everyone. The interview is expensive but tells you who is actually a fit. Reversing the order, doing real interviews with everyone before any skim, would take forever. Retrieval works the same way. A cheap first stage looks at the whole corpus and shortlists the top hundred. A more expensive cross-encoder then reads each shortlisted document together with the query and gives a sharp relevance score. The top ten by that score is what you actually use.

Key concepts

Embeddings

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

60s: bi-encoder retrieves top-100 fast via ANN; cross-encoder rescores those 100 with joint attention to pick top-10 precisely; cost asymmetry is why it must be the second stage; production stack names.

Real-world examples

Real products, models, and research that use this idea.

Cohere rerank-3.5 is the most-used reranker API in 2026 production RAG, paired with Cohere embed-v4 as the canonical bi-encoder.
Voyage rerank-2.5 sits behind Anthropic's contextual retrieval recipe, rescoring Voyage v3 embedding results from the top-100 to the top-10.
BGE-reranker-v2 is the leading open-weight reranker, drop-in compatible with BGE-M3 embeddings for self-hosted RAG.
Jina rerank-v2 ships in the LlamaIndex and LangChain rerank modules with a documented latency budget of ~200ms per 100-candidate rerank.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the cross-encoder typically too expensive to use as the first-stage retriever?

QHow does the choice of top-K for the bi-encoder stage interact with cross-encoder quality?

QWhen would you replace the cross-encoder rerank with a late-interaction retriever like ColBERT?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing a cross-encoder with a bi-encoder. A bi-encoder embeds query and doc separately and compares with cosine. A cross-encoder reads both together in one forward pass.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.