Pick the property that distinguishes dense embeddings from one-hot vectors
Dense embeddings encode semantic similarity geometrically; one-hot vectors are mutually orthogonal and carry no inter-token similarity signal at all.
Picture a row of school lockers. A one-hot label is like saying 'this is locker number 47'. Every locker is its own slot and locker 47 has nothing to do with locker 48, even if both belong to twins. A dense description is more like a seating chart in the cafeteria, where friends sit near each other and strangers sit far apart. The locker system tells you which one a kid owns. The seating chart tells you who likes whom. Both are ways to refer to people, but only the seating chart lets you guess relationships by looking at how close two seats are. That is the difference: locker numbers identify, seating charts relate.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question tests whether you can separate the defining property of an embedding from its incidental characteristics. Three of the four options describe true differences between one-hot vectors and dense embeddings, but only one of them captures why embeddings exist as a concept at all.
The rest are real but secondary: floats vs integers, lower dimensionality, when computation happens. Mistaking any of those for the core difference is exactly the kind of slip an interviewer is probing for.
The one-hot construction in detail
Given a vocabulary V of size N, the one-hot encoding of token t_k is the standard basis vector e_k: a vector of length N with a 1 in position k and zeros everywhere else. This is the canonical way to represent categorical variables for input to a neural network.
The geometry is determined entirely by the construction. For any two distinct tokens t_i and t_j:
Every pair of distinct one-hot vectors has zero dot product and zero cosine similarity. There is no concept of "close" tokens: the vectors are mutually orthogonal, by definition.
This is a feature in some contexts (you want categorical inputs to be linearly independent) and a fatal limitation in others (you want "cat" and "dog" to share some signal). The choice between one-hot and dense is the choice between these two regimes.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Modern embedding APIs like OpenAI text-embedding-3-large and Voyage v3 return dense float vectors, never one-hot.
- The first layer of any transformer LM is an embedding lookup that converts one-hot token IDs into dense vectors: the conversion is itself the point.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat does the matrix-vector product of an embedding matrix W with a one-hot vector e_k actually compute?
It selects column k of W. That's why embedding lookups are mathematically equivalent to a matrix multiply on a one-hot: they're a specialised case. The optimised lookup just skips the wasted zero-multiplications.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking "lower dimensionality" as the defining property. Smaller size is a consequence of dense encoding, not the thing that makes embeddings useful.
60 second bullets to scan on the way to the call.
Define what a one-hot vector is mathematically.
State why one-hot vectors are mutually orthogonal.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.