Flashcard: what is an embedding in a RAG pipeline, and what role does it play?
An embedding is a fixed-length vector that places text at a point in high dimensional space, so semantic similarity becomes vector distance.
Imagine every sentence in the world gets a numeric address, like coordinates on a giant map. Sentences with similar meaning live close together, even when they use different words. 'Cancel my subscription' and 'how do I quit my plan' end up as next door neighbors, while 'best pasta recipes' lives in a completely different neighborhood. A special address maker computer reads each sentence and writes down its address. In RAG, you give every chunk of your documents an address ahead of time and store all the addresses. When a user asks something, you compute the address of their question, then look up which document chunks live nearest. You are no longer searching for matching words. You are searching for matching meaning, which is the whole reason RAG can answer paraphrased questions at all.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Of all the vocabulary in RAG, the word embedding is the one most candidates can define superficially and still get wrong in practice. They say 'a vector representation of text' and stop, which sounds right and explains nothing. An embedding is a very specific object: it is the output of a particular kind of model, it has a fixed dimensionality, it lives in a learned geometry, and it is computed by a process that runs entirely outside the chat LLM.
Getting this picture right matters because the embedding is the substrate of the whole retrieval stage. If your mental model is hazy, you will make confident sounding but wrong decisions: mixing embedding models across the index, treating reindexing as optional, or expecting the generator LLM to 'understand' the vectors directly. None of those work.
This deep dive defines embeddings precisely, walks through the index-time and query-time roles they play in a RAG pipeline, names the 2026-current models you should know, and surfaces the boundary rules that every senior interviewer expects to hear named out loud.
What an embedding actually is
An embedding is the output of an embedding model when you feed it a piece of text. The output is a vector of floating point numbers with a fixed length set by the model: 768, 1024, 1536, 3072, and so on. Every input gets a vector of the same length, no matter how long the text is. A one word query and a 500-token passage both come out as the same shape.
The shape is the easy part. The interesting part is the geometry. The embedding model has been trained, usually with a contrastive objective on pairs of related texts, so that semantically similar inputs land near each other and unrelated inputs land far apart. Distance is typically measured by cosine similarity, which is the angle between two vectors viewed from the origin. The result is a learned high dimensional space where 'cancel my subscription' and 'end my plan' end up as next door neighbors because they meant the same thing in millions of training pairs, even though no two words overlap.
Three things are NOT true about embeddings. They are not part of the chat LLM's weights; they come from a separate model. They are not adjusted at query time; they are precomputed and looked up. And they are not unique per provider; many embedding models exist, and their spaces are incompatible with one another.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI text-embedding-3-large producing 3072-dimensional vectors for RAG corpora over private documents and knowledge bases.
- Voyage Embed v3 asymmetric models offering separate query and document encoders tuned for retrieval, used inside Anthropic's RAG stack.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy must query embeddings and document embeddings come from the same model?
Different models learn unrelated geometries; cosine distance only carries meaning inside a single space. Show what happens when you mix spaces.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating an embedding as part of the LLM's weights. It is a separate artifact computed by a separate model and stored ahead of time, not something the chat model produces during generation.
60 second bullets to scan on the way to the call.
What an embedding is and what its shape looks like
How indexing-time and query-time embeddings are produced
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.