Given a recall@10 budget and a storage budget, pick the right Matryoshka truncation level

MCQ·Hard·4.0 · 0·~1 min·Asked atCognizant Copy Ai Samsung

Attempt it

**The right truncation dimension** under a storage budget and a recall floor is the smallest d that clears the recall floor on your own labeled eval set. That is it. The procedure is empirical and the answer is workload-specific. Storage scales linearly with d. A million 3072-dim float32 vectors take 12 GB; the same million at 768 dims take 3 GB. The 4x ratio is exact, not approximate. Search compute on a dot-product index also scales linearly with d, so a smaller d gives proportional latency improvements as well. The systematic procedure: build an eval set of a few hundred labeled queries from your workload. Truncate the stored vectors to each candidate dim in a ladder like `{128, 256, 512, 1024, 1536, 3072}`. Run retrieval at each dim and measure recall@10. Plot recall against storage cost. Pick the smallest d above your recall floor with a small safety margin. The wrong answers fail in instructive ways. 'Always 256' is a guess that happens to be right for some corpora and wrong for others. 'Half the original' is arbitrary. There is no theoretical reason to pick 1536 over 1024 or 768 absent measurement. 'Matryoshka is marketing' is just incorrect; OpenAI ships it under the `dimensions` API parameter and several open models do too.

**Truncate to the smallest d that meets your recall@10 floor on a labeled eval set.** Storage savings are linear in d. A 4x truncation gives exactly 4x storage savings, and search compute on a dot-product or HNSW kernel also scales roughly linearly with d. The procedure is a Pareto sweep over the cost-quality frontier. Build a labeled eval set drawn from your actual workload (a few hundred queries with at least one known-good doc each is enough for stable recall@10 measurement). Truncate the stored vectors to a geometric ladder of dimensions, typically `{128, 256, 512, 1024, 1536, 3072}` for a model with native dim 3072. Run retrieval at each level. Plot recall against storage. Pick the smallest d above your floor with a margin to absorb measurement variance. The distractors fail in instructive ways. 'Always 256' is a corpus-specific result repackaged as a universal rule. It works for short-query broad-domain workloads and fails for multilingual or technical corpora. 'Half the original' has no theoretical basis. There is no reason for the elbow to sit at exactly d/2. 'Matryoshka is marketing' ignores that OpenAI ships it on text-embedding-3-* via the `dimensions` parameter, and Nomic and Snowflake Arctic Embed do similarly. Several production deployments report 4-8x storage savings at 1-3 point recall cost. The principle: Matryoshka exists to make d a deploy-time decision per workload. Honoring the design means sweeping against the constraints you actually face.

TL;DR

Truncate to the smallest d that meets your recall@10 floor on a labeled eval set, because storage and search compute both scale linearly with d.

Memory aid

Easy to grasp

Imagine you have a strict luggage weight limit and a list of clothes you'd like to bring. You don't pick what to leave out by guessing; you weigh each candidate set on the bathroom scale until you find the lightest combination that still covers what you actually need to wear. Picking a Matryoshka dimension under a storage and recall budget is the same. You weigh candidate dimensions on your own labeled eval set, see which ones still meet your recall requirement, and pick the lightest one that passes. Storage scales straight with dimension, so 'lightest' really does mean 'smallest dim'. Rules of thumb fail because every traveler has different luggage and different weather.

Key concepts

Embeddings Vector Databases

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Picking a Matryoshka truncation level under a storage budget and a recall floor is the canonical production embedding decision. It is also one of the few questions where the wrong answer is recognizable from the way it is framed: any universal recommendation ('always 256', 'half the dim', 'just use the full size') is wrong, because Matryoshka exists precisely to make the decision workload-specific.

The right procedure is a constrained empirical optimization. Storage cost is your objective; recall@10 floor is your constraint; the decision variable is d. The procedure is a sweep over a geometric ladder of dimensions, a measurement of recall on a labeled eval set drawn from your workload, and a pick of the smallest d that clears the floor with a small safety margin.

This answer walks through the linear scaling that makes the objective trivial, the empirical sweep that makes the constraint measurable, and the failure modes of each distractor. The intent is to give a senior engineer the framing and the operational details to defend the choice in a design review.

Why storage and compute both scale linearly with d

The objective in this problem is storage. Storage cost for a corpus of N documents at dimension d using float32 is exactly:

\text{bytes} = N \cdot d \cdot 4

The relationship is exact, not approximate. A million 3072-dim vectors take 12.3 GB; truncate to 768 and you get 3.07 GB. The 4x ratio is the same 4x ratio you see in the dim ladder. There is no hidden overhead per dim, so storage savings track truncation savings one for one.

The search side mostly follows. For dot-product or cosine queries on a brute-force or IVF-flat index, compute is exactly N·d multiplies per query, so search compute and storage move together. On HNSW the relationship is less linear because graph traversal dominates the cost, but the per-comparison cost still scales linearly with d, and the graph build cost scales linearly with d times N, so the operational picture remains the same: smaller d means faster queries, smaller indexes, and less memory pressure.

Query-side bandwidth also matters. If you transmit query embeddings between services, a 4x truncation reduces network bytes by 4x. This is rarely the dominant cost, but in high-QPS systems with internal embedding microservices, it adds up.

The linear scaling is what makes the constrained optimization simple. Every step down the ladder gives you a predictable, exact storage saving. Your only question is how much recall you give up at each step, and that requires measurement.

The sweep procedure in detail

Why each distractor fails

What changes the elbow, and when to re-sweep

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI text-embedding-3-large exposes the `dimensions` API parameter for exactly this trade and documents the cost-quality curve in its launch notes.
Nomic Embed v1.5 ships truncation-quality curves down to 64 dims so users can pick d empirically; production case studies report 4x storage savings at 1-2 recall@10 points cost.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you set the safety margin above the recall floor?

Bootstrap the eval-set metric to get a confidence interval; pick a d whose lower confidence bound is above the floor. A 2-3 percentage point margin handles typical eval-set noise plus modest corpus drift.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating the dimension choice as a model property rather than a deploy-time decision under YOUR constraints. The right d depends on your recall floor, your storage cap, and your corpus; only a sweep tells you.

60 second bullets to scan on the way to the call.

How do storage and search compute each scale with the chosen dimension?
How would you frame truncation as a constrained-optimization problem?

Primary sources. Browse if you want the original framing.