An embedding is a learned dense vector of fixed length whose position in space encodes semantic content: proximity equals similarity.
Imagine every sentence gets pinned onto a giant map. Sentences that mean similar things get pinned close together; unrelated ones land in different neighbourhoods. "The puppy is barking" and "my dog is loud" sit next to each other; "the cake is tasty" sits across town. Once everything has a pin, comparing meanings is just measuring how far apart the pins are. The clever part isn't the map; it's the program that chose the pin locations. It read enormous amounts of text and learned to place new pins so neighbours actually share meaning. After that, sorting, searching, and grouping text becomes simple distance math instead of guessing what words mean.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Most candidates fumble the embedding definition by leading with the use case: "embeddings let you do semantic search." That's a downstream consequence, not a definition. An interviewer wants to know whether you can describe the data object on its own terms and then connect it to the property that makes it useful.
This deep dive unpacks three layers: what the vector is, where the geometry comes from, and why two seemingly innocent shortcuts ("embeddings are compressed text" and "every dimension is a feature") break the definition in subtle ways.
The vector as an object
An embedding is, mechanically, an array of floats with a fixed length. For modern 2026 text embedding APIs, that length is anywhere from 256 (small/efficient) to 3072 (large/high-quality). The model has a fixed output dimensionality, so every input (a one-word query, a 500-token passage, an image) produces a vector of exactly the same shape.
The components are dense floats, not sparse counts or binary flags. That density matters because it's what lets information about meaning be distributed across all coordinates rather than concentrated in a few. A bag of words sparse vector might have 50,000 dimensions with five nonzeros; an embedding has 1024 dimensions all carrying signal.
The object is also self-contained: you don't need a vocabulary, a tokenizer, or the original input to do math on it. Once you have the vector, similarity comparisons, clustering, and ANN search become pure linear algebra. That portability is part of what makes embeddings so useful as an interchange format between systems.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's text-embedding-3-large emits 3072-dimensional dense vectors used in default RAG stacks across 2026.
- Voyage v3 and Cohere embed-v4 are production text embeddings shipped via API, both built on transformer bi-encoders.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat property does the training objective enforce that makes the geometry meaningful?
Contrastive losses (InfoNCE, MNRL, triplet) maximise similarity for related pairs and minimise it for unrelated pairs. The geometry emerges because the loss surface penalises any arrangement where unrelated items sit close.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Defining an embedding by listing what it's used for (search, RAG, clustering) rather than what it IS: a fixed-length dense vector with learned geometric structure.
60 second bullets to scan on the way to the call.
State what an embedding IS as a data object before what it does.
Name the three pieces: fixed dimensionality, dense components, learned geometry.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.