Describe what a 'bi-encoder' is in embedding based retrieval

A **bi-encoder** is a retrieval architecture where the query and each document are encoded **independently** by an embedding model. Each produces its own vector. To compare them, you take the dot product or cosine similarity of the two vectors. The critical operational property: because the document encoding doesn't depend on the query, you can **pre-compute and index** document vectors ahead of time. At query time, you only need to encode the query (one forward pass) and then do a vector-DB lookup against the pre-computed index. That's why every modern vector search system uses bi-encoders. Contrast with **cross-encoders**, where the query and document are concatenated into a single input and run through a joint forward pass. The model can attend across query and document tokens directly, producing more accurate similarity scores, but the computation can't be pre-cached. Because the document representation depends on the specific query. Every query-document pair requires its own forward pass. This is too slow to score millions of documents per query. The practical pattern in modern RAG. Bi-encoder for first-stage retrieval (recall top-100 from millions of documents in tens of milliseconds), cross-encoder reranker for the top-100 (improve precision at the cost of much higher per-pair compute).

A **bi-encoder** encodes the query and the document independently: two separate forward passes producing two vectors, and computes their similarity via dot product orcosine. The query encoder and document encoder are typically the same network (shared weights), though they can differ when symmetric and asymmetric encoders are needed. The key structural property is **independence**. The document encoding is computed without seeing the query, so it can be pre-computed at index time and stored in a vector database. At query time, only the query needs to be encoded (one forward pass), followed by a nearest-neighbour lookup against the index. Latency is dominated by the ANN search (typically 10-50 ms over millions of vectors) rather than by encoding. Contrast with **cross-encoders**, which concatenate query and document into one input `[CLS] query [SEP] document [SEP]` and run a joint forward pass. The model attends across both, producing more accurate similarity (the model can look at "this word in the query corresponds to that word in the document"). The cost: every query-document pair requires its own forward pass, so scoring 1M documents needs 1M forward passes per query. Untenable for first-stage retrieval. The production pattern is **two-stage**: bi-encoder retrieves the top-100 candidates from millions of documents (fast, indexable), then a cross-encoder reranks the top-100 for better precision (slow, accurate). Both stages contribute: the bi-encoder ensures the right documents are in the candidate set; the cross-encoder ensures the right documents are at the top of the final ranking.