Walk through sharding a vector database from 50 million toward 500 million vectors

Short answer·Hard·4.0 · 0·~3 min·Asked atMicrosoft Databricks Google

Attempt it

Your single-node vector database is at 50 million vectors with degrading p99 latency, and the roadmap points at 500 million. Walk through how you would shard it: the partitioning choice, how queries execute across shards, and what gets worse the moment you go distributed.

Free · 2 AI evals / day

Start with the memory math because it forces the decision. At 500 million vectors of 1,536 dimensions at 4 bytes each, the raw storage is around 3 TB before index overhead. No single node holds that comfortably, so you have to shard. The partitioning choice depends on how queries are shaped. If queries are naturally scoped to a tenant, namespace, or product area, shard on that key. Each query touches one shard, isolation between tenants is free, and a hot tenant can get its own dedicated shard. If queries are genuinely global (any query can hit any vector), hash-shard the corpus and scatter-gather: fan the query to every shard, take the top-k from each, merge by score at a router. Add replicas per shard for read throughput and availability. A routing layer handles health checks and load balancing. Things that get worse: p99 latency is now governed by the slowest shard on every scatter-gather query. Recall becomes a per-shard tuning problem. Rebalancing when you add or remove shards is painful unless you planned for it. Hybrid search (vector plus keyword) gets harder because keyword scores are not comparable across shards without global statistics.

**The memory math forces the move and the partitioning choice flows from the query pattern.** Storage at scale: 500M vectors at 1,536 dimensions at 4 bytes is ~3 TB raw, plus ANN graph overhead. One node will not hold it, and even if it could, p99 latency degrades as the index grows because of memory pressure and cache locality loss. Two partitioning strategies. Tenant or namespace sharding when queries are naturally scoped: each query touches one shard, isolation is structural, hot tenants can be moved to dedicated shards. Hash sharding plus scatter-gather when queries are global: fan to every shard, take top-k from each (with k larger than the user-facing k to avoid starvation), merge by score at a router. Replicas per shard handle read throughput and availability. A routing layer handles health checks, load balancing, and request fanout. Things that get worse with distribution: - Tail latency: p99 on scatter-gather queries equals the slowest shard's p99. Mitigations include hedged requests (fire to two replicas, take the first), tight per-shard timeouts, and active shard load balancing. - Recall tuning: per-shard ANN parameters (ef, nprobe) and the k per shard must be set so the merged top-k is not under-served. Wrong tuning silently drops recall. - Rebalancing: adding shards means moving or rebuilding indexes. Plan consistent hashing or logical shards over physical nodes from day one. - Hybrid search: BM25-style keyword scores are not directly comparable across shards without global statistics, complicating fusion. Operationally, per-shard observability and a reindexing runway are mandatory because embedding model migrations now multiply across every shard.

Quick reference

TL;DR

Shard by tenant if queries are scoped, hash and scatter-gather if global. Tail latency, per-shard ANN tuning, rebalancing, and hybrid score fusion all get worse the moment you go distributed.

Memory aid

Easy to grasp

Imagine a library that outgrew one building. You can split it two ways. Option one: give each company its own building. Most visitors only need their company's books, so they go to one building and life is fast. Option two: spread all books across many buildings randomly. Now every visitor sends helpers to all buildings, each helper grabs the best matches, and they meet at the door to pick the overall best. This is slower because the slowest helper sets the pace. The trick is choosing which split matches how people actually search, and then accepting that p99 latency, recall tuning, and rebalancing all get harder no matter which choice you make.

Key concepts

LLM System Design Vector Databases

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

8 min: memory math forces the move, choose partitioning from the query shape, plan for slowest-shard tail latency, tune per-shard ANN and k, plan rebalancing and reindexing from day one.

Real-world examples

Real products, models, and research that use this idea.

Pinecone serverless and pod-based architectures expose namespace and metadata partitioning specifically for tenant sharding
Weaviate and Qdrant support multi-tenancy with per-tenant collections that map cleanly to scoped queries
Milvus and Vespa support scatter-gather across segments for genuinely global queries, with per-segment recall tuning
AWS OpenSearch with k-NN and Azure AI Search both implement sharded vector indexes with router-side merging
Chroma and pgvector at scale typically lean on the underlying storage engine (Postgres, etc.) for sharding rather than building it themselves

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you handle a hot tenant in a tenant-sharded layout?

QWhat does per-shard recall tuning look like in practice?

QHow do you do hybrid search across shards safely?

QWhat does adding a shard look like operationally?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Hash-sharding when queries are tenant-scoped, which forces scatter-gather across every shard for what could have been a one-shard lookup, paying tail latency for no benefit.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.