Pick the property that distinguishes dense embeddings from one-hot vectors

Attempt it

Quick reference

TL;DR

Dense embeddings encode semantic similarity geometrically; one-hot vectors are mutually orthogonal and carry no inter-token similarity signal at all.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a row of school lockers. A one-hot label is like saying 'this is locker number 47'. Every locker is its own slot and locker 47 has nothing to do with locker 48, even if both belong to twins. A dense description is more like a seating chart in the cafeteria, where friends sit near each other and strangers sit far apart. The locker system tells you which one a kid owns. The seating chart tells you who likes whom. Both are ways to refer to people, but only the seating chart lets you guess relationships by looking at how close two seats are. That is the difference: locker numbers identify, seating charts relate.

Key concepts

Embeddings

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Define the one-hot construction. Show all pairs are orthogonal. Contrast with dense embeddings whose geometry is learned. Eliminate the three distractors with one sentence each. Close with the practical reason this matters: retrieval breaks under one-hot.

Real-world examples

Real products, models, and research that use this idea.

Modern embedding APIs like OpenAI text-embedding-3-large and Voyage v3 return dense float vectors, never one-hot.
The first layer of any transformer LM is an embedding lookup that converts one-hot token IDs into dense vectors: the conversion is itself the point.
BGE-M3 and Cohere embed-v4 demonstrate that contrastive training on related pairs is what installs the cosine-similarity property.
BM25 retrieval over an inverted index is closer in spirit to one-hot: it has no inter-term semantic generalisation, which is why hybrid dense+BM25 retrieval often wins.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat does the matrix-vector product of an embedding matrix W with a one-hot vector e_k actually compute?

Sign in or go Pro to see the approach for this follow-up.

QCould you construct a one-hot variant that has some semantic structure?

Sign in or go Pro to see the approach for this follow-up.

QIf dense embeddings are better, why do BM25 / sparse retrievers still exist in production?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Picking "lower dimensionality" as the defining property. Smaller size is a consequence of dense encoding, not the thing that makes embeddings useful.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.