You are designing vector search for a B2B platform: 100M vectors total, 6,000 tenants, hard isolation requirements. One whale tenant owns 40M vectors; the median tenant owns under 5,000. Queries are always single tenant. Design the index layout and explain how you handle the skew, noisy neighbor risk, and per tenant operations like offboarding.
Tier tenants by size: dedicated infra for the 40M whale, packed shared partitions for the long tail, and a routing catalog that promotes tenants as they grow.
Imagine running a storage facility where one company rents 40 percent of all the warehouse space and six thousand small businesses each rent a closet. You would never give every customer the same loading bay. You give the giant tenant their own building with its own forklifts, you give the closet tenants a shared room with labeled shelves, and you put medium customers in a normal warehouse with assigned aisles. A clipboard at the front desk tells the driver which building to visit for which tenant. If a closet tenant grows into a real business, you move them to a bigger room overnight. When someone leaves, you empty their shelves cleanly instead of leaving boxes behind.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The instinct in multi-tenant vector search is to build one big index and add a tenant-id filter. It looks elegant in a design doc and survives the first hundred tenants. Then you onboard a tenant that owns forty percent of the corpus, and the system falls over.
This question is really about heavy-tailed tenant distributions. When one tenant is ten thousand times bigger than the median, no single index configuration is correct for both. The whale wants quantization, dedicated nodes, and careful sharding. The tail wants packing efficiency and almost no per-tenant index overhead. A uniform layout serves neither well and lets the whale's working set starve everyone else.
The production answer is a placement layer: tier the tenants by size, route every query through a catalog, and treat tier promotion as a standard migration. The rest of this section walks through each layer and the failure modes that motivate the design.
Why one shared HNSW does not work
A global index with a tenant-id filter looks simple, but several things go wrong at scale.
First, the whale's vectors dominate the graph. HNSW search latency depends on the structure of the visited neighborhood, and a graph where forty percent of nodes belong to one tenant means most random walks pass through whale-only regions. For non-whale queries, the engine wastes traversal budget on nodes it must filter out. Filtered ANN is fundamentally less efficient than unfiltered ANN, and at low selectivity the gap is steep.
Second, the working set is wrong. Page cache, OS file cache, and the engine's internal buffer pool are all sized assuming uniform access. The whale's hot vectors evict everyone else's. p99 latency for the tail collapses not because the index is slow, but because every query starts cold.
Third, operational events become cross-tenant incidents. Compaction jobs triggered by the whale's ingest pause everyone. A bulk delete of a churned whale account leaves tombstones that degrade neighbor tenants' recall for weeks (see the bulk-delete recall-drop question). One tenant's request to a regulator becomes everyone's outage.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Pinecone serverless uses namespaces as the per-tenant partition and prices dedicated pods separately for whale workloads
- Turbopuffer is built around exactly this whale-vs-tail split: object-storage-backed namespaces for the tail, dedicated tier for hot tenants
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the latency cost of consulting the routing catalog on every query, and how do you hide it?
Catalog reads are cacheable and rarely changing. Cache the tenant-to-placement map at the API gateway with TTL plus invalidation, so the hot path never round-trips to Postgres.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forcing all 6,000 tenants into one global index with a tenant-id filter, then watching the whale starve every other tenant of cache and recall.
60 second bullets to scan on the way to the call.
When to give a tenant its own collection versus a shared namespace
Why exact search beats ANN for tiny tenants
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.