You're standing up an embedding based retrieval system over 50,000,000 documents on a single node with 10 GB of available RAM. You need to fit the vectors in memory (HNSW requires this) with some headroom for graph edges. Walk through how you'd pick the embedding dimension and what compromises you make.
Combine Matryoshka truncation with binary or int8 quantization, size the result against an HNSW 2x overhead, then validate recall on a labeled set.
Imagine you have a small suitcase and far too many shirts. You have three tricks. You can pick fewer shirts. You can roll each shirt tighter. You can squash them into a vacuum bag. One trick alone is not enough to close the lid. Combine two of them and the suitcase finally shuts. Then you take the shirts out, wear them, and check whether they still look good before you head to the airport. Otherwise you arrive at the wedding wearing wrinkled rags. Same with packing your search index. Pick the cleanup steps, stack them carefully, then double check the outfits look right before the big trip.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This is a classic capacity question. The trap is to anchor on a familiar dim (1536) without doing the storage arithmetic. The real work is reasoning across three levers (dim, per-dim precision, and graph architecture) and landing on a configuration that fits the budget while staying within an acceptable recall envelope.
The walkthrough below does the math, names the levers, picks a defensible configuration, and ends with the validation step that turns a guess into an engineering decision.
The storage arithmetic
Raw float32 baseline
For 50M docs at d dimensions in float32:
At d = 1024 that is 205 GB. At d = 256 it is 51 GB. Both blow the 10 GB budget by an order of magnitude.
Per-precision cost table
| Precision | Bytes per dim | At d=1024, 50M docs | At d=512, 50M docs |
|---|---|---|---|
| float32 | 4 | 205 GB | 102 GB |
| float16 | 2 | 102 GB | 51 GB |
| int8 | 1 | 51 GB | 26 GB |
| binary | 0.125 | 6.4 GB | 3.2 GB |
Only the binary row fits comfortably at d = 1024. Int8 fits only at very low dims that lose too much recall.
HNSW edge overhead
HNSW stores M neighbors per node. At default M = 32 the overhead is roughly 256 bytes per node, or ~13 GB for 50M nodes. That number itself exceeds the budget at this scale before you even count the vectors. Setting M = 16 cuts it to ~6 GB, M = 12 to ~4.5 GB. M trades recall for memory.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Pinecone Serverless 2026 uses Matryoshka truncation + binary quant as a default budget configuration.
- OpenAI text-embedding-3-large at 3072 dims with binary quant via libraries like sentence-transformers + faiss-binary.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere exactly is the 2x HNSW overhead coming from?
Each node stores M edges at the bottom layer and a smaller M0 on upper layers, plus link IDs and neighbor distances. At M=32, that is roughly 256 bytes per node, comparable to a 256-d int8 vector.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that HNSW graph edges roughly double the per-vector footprint, so a 'fits in RAM' calculation that ignores graph overhead misses by 2x.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.