Doubling the embedding dimension from 768 to 1536 — what changes in storage, query latency, and PQ trainability?

MCQ·Medium·4.0 · 0·~1 min·Asked atPinecone·Relevant atCohere Databricks

Attempt it

An embedding dimension `d` controls how many numbers each vector contains. For float32 storage, each vector occupies `4 × d` bytes. Doubling `d` from 768 to 1536 doubles per-vector storage from 3072 bytes to 6144 bytes. **Storage.** A 50M-vector index goes from 150GB at 768 dims to 300GB at 1536 dims. The whole storage axis of the bill doubles. **Per-vector distance work.** Computing a dot product or L2 distance is `O(d)` floating-point operations. Doubling `d` doubles the per-vector compute. Modern CPUs use SIMD (AVX-512 handles 16 float32 multiplies per cycle), which helps the *constant factor*, but the linear scaling in `d` is real: the index that ran at 1ms per query at 768 dims runs closer to 2ms per query at 1536 dims, all else equal. **Product Quantization trainability.** PQ splits each vector into `pq_m` sub-vectors and learns a 256-entry codebook per sub-vector. If you double `d` while keeping `pq_m` fixed, each sub-vector now covers twice the dimensional space, and the 256-entry codebook has to represent that larger subspace as accurately as before. Codebook quality degrades. The standard mitigation is to double `pq_m` so each sub-vector stays the same width. The distractors all conflate something true (curse of dimensionality, SIMD) with the actual math.

**Doubling the embedding dimension is linear in d cost on every axis except a few constant-factor SIMD wins.** **Storage.** `4 × d` bytes per float32 vector. Doubling `d` doubles bytes per vector and therefore doubles the storage bill across the index. For a 100M-vector workload that is the difference between `~400GB` and `~800GB`. **Per-query distance compute.** A dot product or L2 squared distance is `O(d)`. Doubling `d` doubles the multiply-adds per comparison. The index touches a roughly fixed candidate count per query (HNSW: ~`ef_search` candidates; IVF: ~`nprobe × cluster_size`), so total per-query compute roughly doubles. SIMD constants help but do not change the slope. **Memory bandwidth.** Each candidate vector is loaded from RAM to compute the distance. Doubling `d` doubles the bytes per comparison loaded, and on memory bandwidth limited workloads (large indexes, low arithmetic intensity) latency can scale slightly worse than 2x. **PQ trainability.** Product Quantization splits the vector into `pq_m` sub-vectors and learns a 256-codeword codebook per sub-vector. If `pq_m` is fixed, each sub-vector now spans twice the original space and the codebook (still just 256 codes) struggles to cover it with the same fidelity. Recall under PQ degrades. The standard mitigation is to double `pq_m` so each sub-vector covers the same width, but then the compressed vector is also twice as long, partially eating into the compression win. **Distractors worth recognizing.** Option B (SIMD makes the cost flat) is the most tempting wrong answer; SIMD is a constant-factor improvement, not asymptotic. Option C is fiction (vector DBs do not store hashes). Option D conflates the curse of dimensionality (a real but subtle high-dim effect) with the straightforward linear storage math.

Quick reference

TL;DR

Doubling d doubles storage and per-vector distance work (both linear in d) and makes PQ codebooks cover larger sub-vectors; fix by raising pq_m.

Memory aid

Easy to grasp

Think of each item in your database like a postal address written on a strip of paper, where the length of the strip is the dimension. Imagine doubling how long every strip is. Now every strip uses twice the storage in the filing cabinet, and reading a strip to compare it to another takes twice as long. For chunked-codebook shrinking (called PQ, which stores each strip's chunks as numbers from a tiny catalogue), keeping the same number of chunks while doubling the strip length means each chunk now covers twice as many characters, and a longer chunk is harder to summarize well with a small catalogue. The standard fix is to slice the longer strip into more chunks, rather than into the same number of fatter chunks.

Key concepts

Product quantization

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: linear scaling of storage and compute with d, SIMD as a constant factor, memory bandwidth as a separate bottleneck, PQ codebook trainability, and Matryoshka truncation as the production escape hatch.

Side-by-side

Axis	Scaling with d	Mitigation
Raw storage (float32)	Linear (4 × d × N)	Use float16, int8, PQ, or BQ
Distance compute per pair	Linear (O(d))	SIMD reduces constant; slope is fixed
Memory bandwidth per pair	Linear (d × bytes loaded)	Compress vectors to reduce bytes/pair
PQ codebook quality at fixed pq_m	Degrades	Scale pq_m proportionally to d
Curse-of-dimensionality (recall)	Asymptotic, mild at d < 4096	Use natural-language manifold-aware embeddings

Real-world examples

Real products, models, and research that use this idea.

text-embedding-3-large at 3072 dims doubles storage versus text-embedding-3-small at 1536 dims; documented as a top cost surprise on Pinecone Serverless.
Cohere embed-v3 at 1024 dims sits at the lower-cost end; embed-v3-large at 4096 dims is the high-recall option for big budgets.
BGE-M3 supports multiple output dimensions (768, 1024) so teams can pick the cost-quality point at the embedding model layer.
Faiss PQ implementations default to pq_m chosen so that each sub-vector is 4-8 dims for d=128, scaling proportionally for larger d.
Matryoshka representation learning lets you truncate text-embedding-3-large from 3072 to 512 dims at the model level, cutting storage 6x while keeping most of the recall.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does memory bandwidth interact with the dimension scaling on HNSW versus IVF?

QIf you double d but the recall@10 only improves by 0.5 points, is the migration worth it?

QWhy does Matryoshka representation learning let you decouple model size from index size?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming SIMD makes higher-dimension queries free. SIMD helps the constant factor; the linear in d scaling of work and storage is unaffected.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.