Bi-encoders vs cross-encoders, where does attention cross the query and document?

Attempt it

Quick reference

TL;DR

Bi-encoder encodes query and document independently and scores by dot product; cross-encoder concatenates them into one sequence and runs full self-attention across the boundary.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine matching job applicants to job postings. A bi-encoder is like having an interviewer write a one-paragraph summary of each applicant and each job, then matching them by comparing summaries side by side, fast, because you only need to write each summary once. A cross-encoder is like making the interviewer sit down with each applicant-job pair, read both together, and write a fresh assessment for that specific pair, slow, because every pair needs its own assessment, but the assessments are far more accurate. Production search stacks use both: the fast bi-encoder narrows millions of jobs down to a hundred, then the slow cross-encoder picks the best ten.

Key concepts

Bi-encoder Cross-encoder Dense retrieval Reranking

Quick scan · 2–4 min

Everything important, quickly.

7 min: bi-encoder vs cross-encoder structural split, document precomputation property, scoring mechanisms, accuracy vs throughput trade-off, two-stage retrieval pipeline, ColBERT as a middle-ground architecture, hybrid sparse-dense first-stage retrieval.

Side-by-side

Property	Bi-encoder	Cross-encoder
Input structure	Query and document encoded separately	Concatenated into one sequence
Cross-modal attention	None	Full self-attention across the boundary
Scoring	Dot product of pooled embeddings	Classification head on [CLS]
Document precomputation	Yes (one forward pass per document offline)	No (every query-document pair needs fresh pass)
Inference cost per query	1 forward pass + N dot products	N forward passes
Scalability	Billions of documents with ANN index	Thousands at most
Accuracy	Lower (pooled embeddings lose token-level signal)	Higher (full token-level interaction)
Typical role	First-stage recall	Second-stage rerank

Real-world examples

Real products, models, and research that use this idea.

OpenAI text-embedding-3-large and Cohere embed-v4 are bi-encoder embedding models powering first-stage retrieval in 2026 production RAG stacks.
bge-reranker-v2.5 and Cohere rerank-v3 are cross-encoder rerankers used for second-stage precision over top-100 candidates from bi-encoder recall.
Anthropic's tool-augmented Claude Opus 4.7 search uses a bi-encoder dense retrieval plus cross-encoder rerank pipeline for grounded responses.
ColBERT late-interaction sits between bi-encoder and cross-encoder, encoding query and document as per-token embeddings and scoring with MaxSim, used in enterprise search at lower latency than pure cross-encoders.
Llama 4 Maverick's retrieval-augmented serving uses hybrid sparse-dense first-stage retrieval (BM25 + bi-encoder) followed by cross-encoder reranking.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does ColBERT's late-interaction architecture sit between bi-encoder and cross-encoder?

Sign in or go Pro to see the approach for this follow-up.

QWhy does a cross-encoder achieve higher accuracy than a bi-encoder even with the same backbone weights?

Sign in or go Pro to see the approach for this follow-up.

QWhen does hybrid sparse-dense (BM25 + bi-encoder) retrieval beat pure dense retrieval?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.