Bi-encoders vs cross-encoders, where does attention cross the query and document?

MCQ·Medium·4.0 · 0·~1 min·Asked atKore Ai Razorpay Tesla·Relevant atAi4bharat Cerebras Deepseek Microsoft

Attempt it

**The answer is B.** In information retrieval and reranking, two encoder architectures dominate: ### Bi-encoder (also called dual-encoder) - Query and document are encoded **independently** through the same model (shared weights, separate forward passes). - Each encoder produces a pooled embedding (often `[CLS]` or mean-pooled). - The relevance score is the **dot product (or cosine)** of the two embeddings. - **No attention crosses the query-document boundary** because the two never share a sequence. **The big property**: you can precompute every document's embedding once, store it in an ANN index (FAISS, HNSW, ScaNN), and at query time just embed the query once and do dot products against the index. Scales to billions of documents. ### Cross-encoder - Query and document are **concatenated** into one sequence: `[CLS] query_tokens [SEP] document_tokens`. - The whole sequence runs through full self-attention. - Every query token can attend to every document token and vice versa, the boundary is invisible to attention. - The relevance score comes from a classification head on `[CLS]`. **The big trade-off**: every (query, document) pair needs a fresh forward pass. You cannot precompute. So cross-encoders are too slow for first-stage retrieval over millions of candidates. ### Production combines both The canonical RAG retrieval stack: bi-encoder for first-stage recall (millions to thousands), cross-encoder reranker for second-stage precision (thousands to tens). The bi-encoder gives speed; the cross-encoder gives accuracy. ### Why the other options are wrong - **A** flips the architectures (assigns cross-attention to the bi-encoder). - **C** says they are the same architecture, structurally false. - **D** invents a non-existent mechanism (only-`[CLS]` cross-boundary attention).

**B is correct.** A **bi-encoder** runs query and document through the same encoder weights but as separate forward passes. Neither sees the other's tokens. Each produces a pooled embedding `e_q` and `e_d`, and the relevance score is the dot product: $$s(q, d) = e_q \cdot e_d$$ Crucially, you can precompute every document's `e_d` once and store it in an ANN index. At query time you compute `e_q` once and score against the index in milliseconds for millions of documents. A **cross-encoder** concatenates query and document into one sequence: `[CLS] query [SEP] document [SEP]`. Full self-attention runs over the whole thing, so every query token can attend to every document token and vice versa. A classification head on `[CLS]` produces the score. The model captures rich joint interactions but you must rerun the entire model for every (query, document) pair. **Production retrieval typically combines both**: 1. Bi-encoder recall over millions of documents in milliseconds. 2. Cross-encoder reranker over the top 50-100 candidates for precision. The bi-encoder pays a slight accuracy cost for massive throughput; the cross-encoder pays massive throughput cost for accuracy. Combining them is the practical sweet spot. **Why the distractors fail:** - A flips the architectures, the bi-encoder is the one WITHOUT cross-attention. - C confuses architecture with loss function. Both can be trained with various losses; the architecture is what differs. - D invents a mechanism (`[CLS]`-only cross-boundary attention) that does not exist.

TL;DR

Bi-encoder encodes query and document independently and scores by dot product; cross-encoder concatenates them into one sequence and runs full self-attention across the boundary.

Memory aid

Easy to grasp

Imagine matching job applicants to job postings. A bi-encoder is like having an interviewer write a one-paragraph summary of each applicant and each job, then matching them by comparing summaries side by side, fast, because you only need to write each summary once. A cross-encoder is like making the interviewer sit down with each applicant-job pair, read both together, and write a fresh assessment for that specific pair, slow, because every pair needs its own assessment, but the assessments are far more accurate. Production search stacks use both: the fast bi-encoder narrows millions of jobs down to a hundred, then the slow cross-encoder picks the best ten.

Key concepts

Bi-encoder Cross-encoder Dense retrieval Reranking

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Bi-encoders and cross-encoders are the two foundational architectures for neural retrieval, and the distinction between them is one of the most consequential design choices in any RAG or search system. The split is purely structural, where attention crosses (or does not cross) the boundary between query and document, and it determines whether documents can be precomputed and indexed or whether every query-document pair requires a fresh forward pass.

This card walks the two architectures, the cost model each implies, why production stacks combine them in a two-stage pipeline, and the modern variants (ColBERT, hybrid sparse-dense, LLM-as-reranker) that fill the cost-accuracy spectrum between pure bi-encoder and pure cross-encoder. By the end you should be able to design a retrieval stack from scratch and reason about which component runs at which scale.

The structural split

Both architectures use the same transformer encoder backbone (BERT, RoBERTa, ModernBERT, or a modern embedding LLM like E5-Mistral). The split is where attention runs.

Bi-encoder

code

Query     ->  [CLS] q1 q2 ... qm [SEP]  -> Encoder -> pool -> e_q (768 dim)
Document  ->  [CLS] d1 d2 ... dn [SEP]  -> Encoder -> pool -> e_d (768 dim)
Score     =   e_q . e_d

Two independent forward passes through the same encoder weights. No attention ever crosses between query tokens and document tokens because they never share a sequence. The relevance score is a dot product (or cosine, after L2 normalization).

Cross-encoder

code

Input  -> [CLS] q1 q2 ... qm [SEP] d1 d2 ... dn [SEP]
       -> Encoder (full self-attention) -> pool -> [CLS] hidden state
Score  = classification_head([CLS] hidden state)

One sequence containing query and document tokens, separated by [SEP]. Full self-attention runs over the entire sequence, so every query token can attend to every document token and vice versa. The relevance score comes from a classification (or regression) head on the [CLS] hidden state.

The boundary visibility

The word 'boundary' is the key. In the bi-encoder, there is no shared sequence, so there is no boundary for attention to cross. In the cross-encoder, the boundary is the [SEP] token in the middle of the sequence, and attention crosses it freely just as it would cross any other position. The bi-encoder structurally CANNOT model query-document token interactions because the two halves are never in the same forward pass.

Cost model: why one scales and the other doesn't

The two-stage retrieval pipeline

Modern variants on the cost-accuracy spectrum

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	Bi-encoder	Cross-encoder
Input structure	Query and document encoded separately	Concatenated into one sequence
Cross-modal attention	None	Full self-attention across the boundary
Scoring	Dot product of pooled embeddings	Classification head on [CLS]
Document precomputation	Yes (one forward pass per document offline)	No (every query-document pair needs fresh pass)
Inference cost per query	1 forward pass + N dot products	N forward passes
Scalability	Billions of documents with ANN index	Thousands at most
Accuracy	Lower (pooled embeddings lose token-level signal)	Higher (full token-level interaction)
Typical role	First-stage recall	Second-stage rerank

Real products, models, and research that use this idea.

OpenAI text-embedding-3-large and Cohere embed-v4 are bi-encoder embedding models powering first-stage retrieval in 2026 production RAG stacks.
bge-reranker-v2.5 and Cohere rerank-v3 are cross-encoder rerankers used for second-stage precision over top-100 candidates from bi-encoder recall.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does ColBERT's late-interaction architecture sit between bi-encoder and cross-encoder?

ColBERT encodes query and document into per-token embeddings (not pooled), so storage is more expensive than bi-encoder. Scoring uses MaxSim: for each query token, find the max similarity to any document token, then sum. This captures some cross attention like interaction without rerunning the model per pair. Documents can be precomputed (storing per-token embeddings) and queries score against them with MaxSim, sub-quadratic cost. Accuracy lands between bi-encoder and cross-encoder, latency lands closer to bi-encoder.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Mistaking the bi-encoder for using cross-attention between query and document. Bi-encoders run independent forward passes with zero cross-modal attention; the dot product between pooled embeddings is what couples them at scoring time.

60 second bullets to scan on the way to the call.

Describe the bi-encoder architecture and how it produces a score
Describe the cross-encoder architecture and what it concatenates

Primary sources. Browse if you want the original framing.