How does ColBERT differ from pooled-vector retrieval?

**ColBERT** is a retrieval architecture that takes a different shape from standard sentence embeddings. A normal bi-encoder runs a transformer over each document and pools the token vectors into one fixed-dim vector per document. ColBERT skips the pooling and stores every token's vector independently. The scoring works differently too. At query time the query is tokenized and embedded the same way (one vector per query token). The relevance score is called **MaxSim**: for each query token, find the maximum cosine similarity to any document token, then sum those maxima across query tokens. The phrase **late interaction** captures the idea. In a cross-encoder, query and document tokens interact early. They go through the transformer together. In a bi-encoder, they never interact at all. Each side is encoded and pooled independently. ColBERT sits in between: each side is encoded independently (so document encoding is precomputable at index time). The per-token comparison at query time gives some of the fine-grained matching that a cross-encoder provides. The cost is storage. Instead of one vector per document, you store one per token, which is typically 10-50× more data. ColBERTv2 and PLAID reduce that overhead with quantization and clustering tricks.

**ColBERT** keeps one vector per token in each document, rather than pooling them into a single document vector. The scoring function is **MaxSim**: for each query token `q_i`, compute the maximum cosine similarity against any document token, then sum across all query tokens. Formally: $$\text{MaxSim}(q, d) = \sum_{i \in q} \max_{j \in d} \cos(q_i, d_j)$$ This is called **late interaction** because the per-token comparison happens at query time (late in the pipeline), in contrast to a cross-encoder where query and document tokens interact inside the transformer (early), or a bi-encoder where pooled vectors interact through a single similarity score (no token-level interaction at all). The tradeoff. Quality: ColBERT-style models typically beat bi-encoders by 5-15 points on BEIR because fine-grained term-level signal is not washed out by pooling. Storage: each document costs roughly 10-50x a pooled bi-encoder embedding, because you store one vector per token instead of one per document. Latency: per-token MaxSim is slower than a single ANN lookup, though PLAID's clustering brings it close. Variants. ColBERTv2 quantizes vectors and centroids to reduce storage. PLAID further organizes the per-token vectors into a clustered index for faster retrieval. ColBERT-XS shrinks the per-token dimension. The 2026 production use case is mainly quality-critical RAG on mid-sized corpora; large-scale stacks still default to bi-encoder + cross-encoder rerank because the storage cost is steep.