A search team wants to upgrade from single vector retrieval (one 1024-dim float32 embedding per document) to ColBERT style late interaction, which stores one 128-dim float32 vector per token. Documents average 100 tokens after truncation. Predict the per document storage multiplier for the vector data, and name the standard mitigation that makes multi-vector deployable at scale.
12.5x storage blowup (51,200 vs 4,096 bytes per doc); ColBERTv2 and PLAID cut it back with roughly 2-bit residual quantization plus token pooling and pruning.
A single-vector model summarizes a whole document as one point in space. ColBERT instead drops a point for every token, then matches queries by comparing each query point to the best document point. That is more expressive, but if your document is 100 words long, you are now storing 100 points instead of one. Even though each point is smaller (128 numbers instead of 1024), the total still grows by about an order of magnitude. The fix is to squish each point hard, store it in roughly two bits per dimension instead of 32, and throw away the points that do not carry useful matching signal. With those tricks, the storage cost lands close to the original single-vector design while keeping most of the quality gain.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Late interaction retrieval, popularized by ColBERT, gives a measurable quality lift over pooled single-vector retrieval on many benchmarks. The catch is that 'one vector per token' is a deceptively expensive design decision. The 100-token average per document in this scenario is realistic for many domains (paragraphs, abstracts, support tickets) and produces a 12.5x storage blowup on its face, plus a 100x index-entry blowup that often surprises teams more than the byte count.
This deep-dive does the math three ways (bytes, entries, scoring cost), then walks through the engineering that makes late interaction tractable in production: residual quantization, centroid-based indexing, token pruning, and the two-stage hybrid pattern that most engines now ship as their default deployment shape.
The arithmetic and the three cost axes
Vector bytes per document
Single vector: 1024 dims * 4 bytes/dim = 4096 bytes.
Multi-vector: 100 tokens * 128 dims * 4 bytes/dim = 51,200 bytes.
Ratio: 51,200 / 4,096 = 12.5x. Notice that the dimension reduction (1024 to 128) only saves 8x; the 100-token multiplication is much larger and dominates.
Index entries per document
This is the cost that surprises teams. An HNSW graph or IVF posting-list index sizes its overhead by the number of indexed vectors, not the byte count. Going from 1 vector per document to 100 multiplies the graph size by 100x, the posting list count by 100x, and the build time by similar.
On a 10M-document corpus: single-vector HNSW has roughly 10M nodes and fits comfortably in tens of GB of RAM. Multi-vector raw HNSW has 1B nodes and does not fit on a single machine. Sharding is forced, latency rises, and the operational cost grows non-linearly.
Scoring cost per query
Single-vector scoring per candidate: one dot product of dimension 1024, roughly 1024 multiply-adds.
Multi-vector MaxSim per candidate: for each of Q query tokens, take the max similarity over D document tokens. For a 10-token query against a 100-token document, that is 10 * 100 = 1000 dot products of dimension 128, each of which is 128 multiply-adds. Total: 128,000 multiply-adds per candidate, roughly 125x more compute than a single-vector dot product. The compute scales linearly in query length, document length, and candidate count.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- ColBERTv2 / PLAID published by the Stanford Stanford-FutureData group quantizes residuals to about 2 bits per dimension to bring storage close to single-vector parity.
- Vespa supports multi-vector documents natively and pairs them with HNSW per token plus aggressive ranking-stage pruning.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does PLAID's centroid-based first stage interact with the residual quantization?
Centroids form an inverted index: each centroid points to documents whose tokens fall in that cell. Query tokens hit a small set of centroids; candidate docs are those with the highest centroid co-occurrence. Only then do you decompress residuals and run MaxSim on the survivors.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that the blowup also multiplies index entries: 100 vectors per doc means HNSW or IVF builds an index 100x larger, not just 12.5x more raw storage.
60 second bullets to scan on the way to the call.
The exact arithmetic: tokens times dims times bytes versus single dim times bytes
Index-entry blowup as a separate cost from vector-byte blowup
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.