Billion-scale credible: Milvus / Zilliz, Vespa, Pinecone enterprise. Not credible: Chroma (embedded), pgvector default (single instance Postgres), LanceDB (analytical workflows, not high QPS retrieval).
Imagine you need a warehouse for a billion small boxes. You would not pick a kitchen pantry, a single room storage unit, or a shed designed for garden tools. You would pick a multi-building logistics facility with forklifts, conveyors, and a staff that has run something this size before. Vector databases work the same way. Some products are designed for billions from day one (Milvus, Vespa, Pinecone enterprise); others are excellent at small scale but were never built to be split across many machines. The question is asking you to tell the categories apart by their architecture, not by their marketing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Billion-vector scale is a useful boundary because it forces architectural choices that simply do not appear at smaller scale. At 2M vectors any modern vector DB works. At 200M vectors a serious subset works. At 2B vectors only a handful of architectures are credible, and the question becomes less about feature lists and more about whether the system was designed for the scale being asked of it.
The 2026 landscape splits cleanly along this line. Three vendors in this question have multi-shard horizontal scaling, compressed indexes, and public deployments at billions. Three do not, for clear architectural reasons that the question is testing whether the candidate can name.
What architectural bar does billion-scale impose?
A vector DB credible at billions needs four properties.
First, multi-shard horizontal scaling. A billion-vector index does not fit in the RAM of any single machine in a useful configuration. The data has to be split across many shards, each holding a slice of the corpus, with query time scatter-gather and merge. Systems without multi-shard architecture in their core design hit a hard wall.
Second, compressed indexes that preserve query performance. Storing billions of float32 vectors at 1024 dimensions is roughly 4TB of raw data; even sharded, this is expensive. The credible billion-scale engines support PQ, BQ, or DiskANN to bring the per vector cost down by 10-100x while keeping recall acceptable.
Third, segment based or LSM-style storage so that writes do not block reads. Monolithic in-memory graphs (vanilla HNSW) struggle under heavy upsert workloads at billions; the credible engines split data into segments, write to new segments, and compact in the background.
Fourth, replication and a real operational track record. Billion-scale workloads run for years. Vendors with public reference deployments at billions have battle tested their architecture in ways that paper benchmarks cannot replicate.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Vendor | Architecture shape | Billion-scale credible? |
|---|---|---|
| Milvus / Zilliz Cloud | Multi-shard, segment based, IVF-PQ + HNSW + DiskANN | Yes, public LinkedIn / Salesforce deployments |
| Vespa | Horizontally scaled IR engine, hybrid sparse + dense | Yes, Yahoo / Spotify deployments |
| Pinecone enterprise | Managed multi-region, compressed indexes | Yes, multi-tenant managed offering |
| Chroma | Embedded in-process library | No, single machine design |
| pgvector default | Postgres extension on single instance | No, peaks around 10-50M vectors |
| LanceDB | Columnar on disk analytical format | No, design center is data lake not online retrieval |
Real products, models, and research that use this idea.
- LinkedIn operates Milvus at billion-vector scale for embedding based recommendations across feed, search, and ads ranking.
- Salesforce Einstein uses Milvus for billion-scale embedding retrieval across customer facing AI features.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does Milvus's segment based architecture help at billion scale compared to a monolithic HNSW index?
Each segment is small and immutable once sealed; updates create new segments; the compactor merges segments in the background. This avoids the monolithic graph rebuild bottleneck and lets reads and writes scale independently across segments and shards.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming any modern vector DB can scale to billions because the marketing says so. Architecture matters: embedded single machine stores and single instance Postgres extensions hit hard walls long before a billion vectors.
60 second bullets to scan on the way to the call.
Architectural bar for billion-scale: multi-shard + compressed indexes + segment based storage + replication
Milvus / Zilliz Cloud as the canonical open source choice
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.