Bi-encoders vs cross-encoders, where does attention cross the query and document?

MCQ·Medium·4.0 · 0·~1 min

Attempt it

**The answer is B.** In information retrieval and reranking, two encoder architectures dominate: ### Bi-encoder (also called dual-encoder) - Query and document are encoded **independently** through the same model (shared weights, separate forward passes). - Each encoder produces a pooled embedding (often `[CLS]` or mean-pooled). - The relevance score is the **dot product (or cosine)** of the two embeddings. - **No attention crosses the query-document boundary** because the two never share a sequence. **The big property**: you can precompute every document's embedding once, store it in an ANN index (FAISS, HNSW, ScaNN), and at query time just embed the query once and do dot products against the index. Scales to billions of documents. ### Cross-encoder - Query and document are **concatenated** into one sequence: `[CLS] query_tokens [SEP] document_tokens`. - The whole sequence runs through full self-attention. - Every query token can attend to every document token and vice versa, the boundary is invisible to attention. - The relevance score comes from a classification head on `[CLS]`. **The big trade-off**: every (query, document) pair needs a fresh forward pass. You cannot precompute. So cross-encoders are too slow for first-stage retrieval over millions of candidates. ### Production combines both The canonical RAG retrieval stack: bi-encoder for first-stage recall (millions to thousands), cross-encoder reranker for second-stage precision (thousands to tens). The bi-encoder gives speed; the cross-encoder gives accuracy. ### Why the other options are wrong - **A** flips the architectures (assigns cross-attention to the bi-encoder). - **C** says they are the same architecture, structurally false. - **D** invents a non-existent mechanism (only-`[CLS]` cross-boundary attention).

**B is correct.** A **bi-encoder** runs query and document through the same encoder weights but as separate forward passes. Neither sees the other's tokens. Each produces a pooled embedding `e_q` and `e_d`, and the relevance score is the dot product: $$s(q, d) = e_q \cdot e_d$$ Crucially, you can precompute every document's `e_d` once and store it in an ANN index. At query time you compute `e_q` once and score against the index in milliseconds for millions of documents. A **cross-encoder** concatenates query and document into one sequence: `[CLS] query [SEP] document [SEP]`. Full self-attention runs over the whole thing, so every query token can attend to every document token and vice versa. A classification head on `[CLS]` produces the score. The model captures rich joint interactions but you must rerun the entire model for every (query, document) pair. **Production retrieval typically combines both**: 1. Bi-encoder recall over millions of documents in milliseconds. 2. Cross-encoder reranker over the top 50-100 candidates for precision. The bi-encoder pays a slight accuracy cost for massive throughput; the cross-encoder pays massive throughput cost for accuracy. Combining them is the practical sweet spot. **Why the distractors fail:** - A flips the architectures, the bi-encoder is the one WITHOUT cross-attention. - C confuses architecture with loss function. Both can be trained with various losses; the architecture is what differs. - D invents a mechanism (`[CLS]`-only cross-boundary attention) that does not exist.

Quick reference

TL;DR

Bi-encoder encodes query and document independently and scores by dot product; cross-encoder concatenates them into one sequence and runs full self-attention across the boundary.

Memory aid

Easy to grasp

Imagine matching job applicants to job postings. A bi-encoder is like having an interviewer write a one-paragraph summary of each applicant and each job, then matching them by comparing summaries side by side, fast, because you only need to write each summary once. A cross-encoder is like making the interviewer sit down with each applicant-job pair, read both together, and write a fresh assessment for that specific pair, slow, because every pair needs its own assessment, but the assessments are far more accurate. Production search stacks use both: the fast bi-encoder narrows millions of jobs down to a hundred, then the slow cross-encoder picks the best ten.

Key concepts

Bi-encoder Cross-encoder Dense retrieval Reranking

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

7 min: bi-encoder vs cross-encoder structural split, document precomputation property, scoring mechanisms, accuracy vs throughput trade-off, two-stage retrieval pipeline, ColBERT as a middle-ground architecture, hybrid sparse-dense first-stage retrieval.

Side-by-side

Property	Bi-encoder	Cross-encoder
Input structure	Query and document encoded separately	Concatenated into one sequence
Cross-modal attention	None	Full self-attention across the boundary
Scoring	Dot product of pooled embeddings	Classification head on [CLS]
Document precomputation	Yes (one forward pass per document offline)	No (every query-document pair needs fresh pass)
Inference cost per query	1 forward pass + N dot products	N forward passes
Scalability	Billions of documents with ANN index	Thousands at most
Accuracy	Lower (pooled embeddings lose token-level signal)	Higher (full token-level interaction)
Typical role	First-stage recall	Second-stage rerank

Real-world examples

Real products, models, and research that use this idea.

OpenAI text-embedding-3-large and Cohere embed-v4 are bi-encoder embedding models powering first-stage retrieval in 2026 production RAG stacks.
bge-reranker-v2.5 and Cohere rerank-v3 are cross-encoder rerankers used for second-stage precision over top-100 candidates from bi-encoder recall.
Anthropic's tool-augmented Claude Opus 4.7 search uses a bi-encoder dense retrieval plus cross-encoder rerank pipeline for grounded responses.
ColBERT late-interaction sits between bi-encoder and cross-encoder, encoding query and document as per-token embeddings and scoring with MaxSim, used in enterprise search at lower latency than pure cross-encoders.
Llama 4 Maverick's retrieval-augmented serving uses hybrid sparse-dense first-stage retrieval (BM25 + bi-encoder) followed by cross-encoder reranking.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does ColBERT's late-interaction architecture sit between bi-encoder and cross-encoder?

QWhy does a cross-encoder achieve higher accuracy than a bi-encoder even with the same backbone weights?

QWhen does hybrid sparse-dense (BM25 + bi-encoder) retrieval beat pure dense retrieval?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Mistaking the bi-encoder for using cross-attention between query and document. Bi-encoders run independent forward passes with zero cross-modal attention; the dot product between pooled embeddings is what couples them at scoring time.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.