Pick the right Matryoshka truncation level given a strict storage budget

MCQ·Medium·4.0 · 0·~1 min·Asked atAirbnb Browserbase Whylabs

Attempt it

**Matryoshka** truncation lets you choose the embedding dimension at deploy time. The question is how to pick the dimension well. The wrong way is to grab a number off a blog post (256, 512, 768) and use it without checking. The right d depends on three things specific to you: how good your embedding model is on your corpus, how strict your quality bar is. How much storage you can afford. None of those are universal, so a universal answer cannot be correct. The right way is an empirical sweep. Build a labeled evaluation set drawn from your workload (a few hundred query-document pairs is enough). Truncate the stored vectors to each candidate dimension in a ladder like `{128, 256, 512, 1024, 1536, 3072}`. Run retrieval at each dim and measure recall@10 (or whatever quality metric matches your product). Plot the curve of recall versus storage cost. Pick the smallest dimension whose recall clears your floor. The curve has a recognizable shape: quality rises sharply at the smallest dims, then flattens. The right d is usually the elbow: the point where adding more dimensions stops paying for itself in quality.

**Sweep, plot, pick.** That is the entire procedure. Build a labeled eval set drawn from your workload: a few hundred queries with known-good document IDs is enough for most retrieval workloads. Truncate the stored vectors to each candidate dimension in a ladder like `{128, 256, 512, 1024, 1536, 3072}`. For each dim, run retrieval against the eval set and measure recall@10, or whatever metric matches your product surface (NDCG@10, MRR, precision@1, whatever your downstream actually cares about). Plot recall@10 on the y-axis and storage cost per million docs on the x-axis. The curve has a predictable shape: steep gains at the smallest dims, then a flattening region where adding more dimensions costs storage without buying quality. The smallest d that clears your recall floor is the right answer. The wrong answers fail in characteristic ways. 'Always use the max dim' wastes storage and ignores the point of Matryoshka. 'Use d=256 as the sweet spot' is a guess that happens to be right for some corpora and wrong for others. 'Pick d as a power of two' confuses index implementation details (no modern vector DB cares about power of two dims) with the actual cost-quality trade. The principle: deploy-time dimensionality is an empirical choice, not a rule of thumb. Storage cost scales linearly with d, quality scales nonlinearly, and the right d is the one your own data tells you.

TL;DR

Sweep a ladder of truncation dimensions on a labeled eval set, plot recall@k versus storage cost, and pick the smallest dimension that clears your quality floor.

Memory aid

Easy to grasp

Imagine you have a giant photo and you need to shrink it to fit your phone wallpaper. You could pick a random small size, or you could try a few sizes side by side and see which one still looks good enough to recognize. Picking the right Matryoshka size for your data is the same. There is no universally correct number, because the right answer depends on your photo and your phone. The honest approach is to try several sizes against your own labeled examples, watch how well people still recognize the picture, and pick the smallest size that still passes your bar. Anyone who tells you 'always use 256' or 'always use the maximum' is guessing.

Key concepts

Embeddings Vector Databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Picking a Matryoshka truncation dimension is one of the rare interview questions where the right answer is a procedure rather than a number. The point of Matryoshka representation learning is to make embedding dimensionality a deploy-time choice; the corresponding production discipline is to make that choice empirically, against your own workload, rather than by rule of thumb.

The four candidate answers in the question reflect the four ways teams typically fail at this decision. 'Always max dim' wastes storage and ignores the design intent. 'd=256 sweet spot' takes a corpus-specific result and rebrands it as a universal rule. 'Power of two for indexes' confuses irrelevant implementation details with the actual cost-quality decision. Only the sweep plot pick procedure adapts to the workload, which is exactly why Matryoshka was designed to support it.

This answer walks through the procedure, the shape of the cost-quality curve, why the distractors fail in instructive ways, and the operational discipline around re-sweeping.

The procedure: sweep, plot, pick the elbow

Step one is building an eval set drawn from your own workload. The minimum useful size is a few hundred queries with at least one known-good document each. For higher-precision regimes (NDCG@1, MRR on long-tail queries) you may need more, but a few hundred is enough to see the shape of the curve. The critical property is that the eval set reflects production traffic: query length distribution, vocabulary, intent mix, and language coverage should all match. An off the shelf benchmark like MTEB is a poor substitute because it measures a different distribution.

Step two is running retrieval at each candidate dimension in a ladder. A good default ladder is the geometric one: {128, 256, 512, 1024, 1536, 3072} for a model with native 3072 dim. The geometric spacing keeps storage costs comparable across steps and surfaces the elbow at whichever d it actually lives.

Step three is the plot. Put the metric on the y-axis (recall@10 is the standard choice for retrieval-as-a-funnel) and storage cost per million documents on the x-axis. The curve has a predictable shape: quality rises sharply at the smallest dimensions, then bends through an elbow, then flattens. The elbow is the regime where the marginal dimension stops buying meaningful quality.

Step four is the pick. Choose the smallest d whose quality clears your floor with a safety margin. The 'safety margin' matters because the eval-set metric is itself a sample estimate of true quality; picking exactly at the floor leaves you exposed to variance.

Document the choice. Future readers (possibly you, six months later) will benefit from knowing what eval set was used, what model version was current, and what the curve looked like at the time of the decision.

Why each wrong answer is wrong

What the curve looks like and what its shape tells you

Operational discipline: when to re-sweep

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's text-embedding-3-large documentation explicitly recommends evaluating the `dimensions` parameter against your own data rather than picking a default.
Nomic Embed v1.5 ships truncation-quality curves in its release notes specifically so users can pick d empirically against their own workload.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow large does the labeled eval set need to be for the sweep to give a stable signal?

A few hundred queries with at least one known-good document each is typically enough for recall@10 to stabilize. Tighter metrics like NDCG@1 need more data. Compute confidence intervals via bootstrap to confirm the curve is stable.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Picking the truncation dim by reputation or round number. The right d depends on your corpus, your quality floor, and your storage budget. Only a sweep tells you the actual trade curve.

60 second bullets to scan on the way to the call.

The empirical-sweep procedure: build eval set, run at each d, plot, pick the elbow
Why no universal optimal dimension exists

Primary sources. Browse if you want the original framing.