Your single-node vector database is at 50 million vectors with degrading p99 latency, and the roadmap points at 500 million. Walk through how you would shard it: the partitioning choice, how queries execute across shards, and what gets worse the moment you go distributed.
Free · 2 AI evals / day
Start with the memory math because it forces the decision. At 500 million vectors of 1,536 dimensions at 4 bytes each, the raw storage is around 3 TB before index overhead. No single node holds that comfortably, so you have to shard.
The partitioning choice depends on how queries are shaped. If queries are naturally scoped to a tenant, namespace, or product area, shard on that key. Each query touches one shard, isolation between tenants is free, and a hot tenant can get its own dedicated shard. If queries are genuinely global (any query can hit any vector), hash-shard the corpus and scatter-gather: fan the query to every shard, take the top-k from each, merge by score at a router.
Add replicas per shard for read throughput and availability. A routing layer handles health checks and load balancing.
Things that get worse: p99 latency is now governed by the slowest shard on every scatter-gather query. Recall becomes a per-shard tuning problem. Rebalancing when you add or remove shards is painful unless you planned for it. Hybrid search (vector plus keyword) gets harder because keyword scores are not comparable across shards without global statistics.
**The memory math forces the move and the partitioning choice flows from the query pattern.**
Storage at scale: 500M vectors at 1,536 dimensions at 4 bytes is ~3 TB raw, plus ANN graph overhead. One node will not hold it, and even if it could, p99 latency degrades as the index grows because of memory pressure and cache locality loss.
Two partitioning strategies. Tenant or namespace sharding when queries are naturally scoped: each query touches one shard, isolation is structural, hot tenants can be moved to dedicated shards. Hash sharding plus scatter-gather when queries are global: fan to every shard, take top-k from each (with k larger than the user-facing k to avoid starvation), merge by score at a router.
Replicas per shard handle read throughput and availability. A routing layer handles health checks, load balancing, and request fanout.
Things that get worse with distribution:
- Tail latency: p99 on scatter-gather queries equals the slowest shard's p99. Mitigations include hedged requests (fire to two replicas, take the first), tight per-shard timeouts, and active shard load balancing.
- Recall tuning: per-shard ANN parameters (ef, nprobe) and the k per shard must be set so the merged top-k is not under-served. Wrong tuning silently drops recall.
- Rebalancing: adding shards means moving or rebuilding indexes. Plan consistent hashing or logical shards over physical nodes from day one.
- Hybrid search: BM25-style keyword scores are not directly comparable across shards without global statistics, complicating fusion.
Operationally, per-shard observability and a reindexing runway are mandatory because embedding model migrations now multiply across every shard.
Quick reference
TL;DR
Shard by tenant if queries are scoped, hash and scatter-gather if global. Tail latency, per-shard ANN tuning, rebalancing, and hybrid score fusion all get worse the moment you go distributed.
Imagine a library that outgrew one building. You can split it two ways. Option one: give each company its own building. Most visitors only need their company's books, so they go to one building and life is fast. Option two: spread all books across many buildings randomly. Now every visitor sends helpers to all buildings, each helper grabs the best matches, and they meet at the door to pick the overall best. This is slower because the slowest helper sets the pace. The trick is choosing which split matches how people actually search, and then accepting that p99 latency, recall tuning, and rebalancing all get harder no matter which choice you make.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Corpus that fits comfortably on one node with headroom Distributed complexity is overhead without payoff at small scale
Workloads where p99 latency is irrelevant and throughput is the only concern A single node with read replicas may be simpler and adequate
Embedding model still volatile and likely to change dimensionality Sharding hardens choices that should remain flexible during research phase
Quick scan · 2–4 min
Everything important, quickly.
8 min: memory math forces the move, choose partitioning from the query shape, plan for slowest-shard tail latency, tune per-shard ANN and k, plan rebalancing and reindexing from day one.
Real-world examples
Real products, models, and research that use this idea.
1Pinecone serverless and pod-based architectures expose namespace and metadata partitioning specifically for tenant sharding
2Weaviate and Qdrant support multi-tenancy with per-tenant collections that map cleanly to scoped queries
3Milvus and Vespa support scatter-gather across segments for genuinely global queries, with per-segment recall tuning
4AWS OpenSearch with k-NN and Azure AI Search both implement sharded vector indexes with router-side merging
5Chroma and pgvector at scale typically lean on the underlying storage engine (Postgres, etc.) for sharding rather than building it themselves
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle a hot tenant in a tenant-sharded layout?
Sign in or go Pro to see the approach for this follow-up.
QWhat does per-shard recall tuning look like in practice?
Sign in or go Pro to see the approach for this follow-up.
QHow do you do hybrid search across shards safely?
Sign in or go Pro to see the approach for this follow-up.
QWhat does adding a shard look like operationally?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Hash-sharding when queries are tenant-scoped, which forces scatter-gather across every shard for what could have been a one-shard lookup, paying tail latency for no benefit.