When people first hear that a sentence becomes a list of 1024 numbers, the natural assumption is that each number measures something: "this one is sentiment, this one is topic, this one is length." That intuition is wrong. Understanding why is fundamental to working with embeddings.
The **dimensions are an arbitrary learned basis**. The training procedure had no instructions about what each coordinate should mean. It just adjusted the model parameters so the right pairs of inputs ended up close in vector space and the wrong pairs ended up far apart.
If you trained the same model again from a different random seed, you'd get a completely different arrangement, but it would still work just as well for similarity search.
Semantic concepts ARE encoded in the vectors. A sentiment classifier trained on top of frozen embeddings DOES find sentiment information. But the sentiment direction is a particular weighted combination of all 1024 dimensions, not any single coordinate. The same is true for topic, length, formality, language, and every other property.
This matters operationally. You cannot "make the sentiment positive" by editing one number in the vector. You cannot "filter by topic" by sorting on a single dimension. The vector has to be treated as an indivisible whole, with manipulations done via vector operations on the full embedding.
**Embedding dimensions don't carry per-axis meaning.** Semantic concepts ARE in the vector, but they live as directions (linear combinations across many or all dimensions) not as individual coordinates.
Three pieces of evidence make this concrete. First, the **random-seed test**: train the same architecture twice with different initialisations and you get two embedding spaces where dimension 47 means completely different things. Both spaces support the same similarity search and downstream classifiers. That alone rules out any built-in coordinate semantics.
Second, **probing classifiers**. When researchers train a linear probe to find sentiment in BERT embeddings, the result is a weight vector across all 768 dimensions, not a one-hot on any single one. Same for topic, syntactic role, language identification.
Third, the **steering literature**: concept editing in LLMs identifies activation-space directions (compute the mean difference between contrasting pairs), and steering applies a multiplier to that direction. Nobody steers by editing dimension 47, because dimension 47 doesn't mean anything in isolation.
The distractors fail. Option A imposes false structure; option C implies per-model human-readable dimensions which still aren't there; option D treats training as encryption, which it isn't.
Quick reference
TL;DR
Embedding dimensions are an emergent learned basis with no inherent meaning; semantic concepts live in directions (linear combinations), not individual coordinates.
Imagine asking ten people to describe a face using exactly one hundred sliders. Each person picks their own meanings for their sliders. One uses brightness, another uses age, another uses kindness. Every face gets a unique slider setting, and faces that look alike end up with similar settings. But slider number forty seven does not mean the same thing to person A as it does to person B. They each made up their own system in private. The number-slots used to describe a piece of text work the same way. The system invents its own slider set during practice. The sliders are not sentiment, topic, or age. They are whatever pattern made the practice scores come out right. Meaning is real but it lives in combinations across many sliders at once, never on any single slider alone.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Building features for an interpretable downstream model. If interpretability matters more than retrieval quality, dense embeddings are the wrong tool. Use structured features or sparse coding instead.
Trying to manipulate a single attribute by editing coordinates. Concepts live in directions, not coordinates. Coordinate edits perturb dozens of entangled concepts simultaneously, producing noise.
Aligning two different embedding models by mapping dimensions one to one. The bases are unrelated. Cross-model alignment requires Procrustes-style transforms or training a projection, not a per-dimension correspondence.
Quick scan · 2–4 min
Everything important, quickly.
State the property: dimensions are an arbitrary basis. Give the random-seed argument. Cite probing classifiers as evidence that concepts ARE encoded, just not on axes. Eliminate each distractor on its specific failure. Close with the practical rule: manipulate via directions, never coordinates.
Real-world examples
Real products, models, and research that use this idea.
1Mechanistic interpretability work on Llama and Claude reveals superposition: concepts spread across dimensions, dimensions across concepts.
2Sparse autoencoders on Anthropic models extract concept-aligned features but only after explicit decomposition. The raw dimensions are not concepts.
3Probing classifiers on BERT, E5, and Voyage embeddings recover sentiment and topic as dense weight vectors across all dimensions.
4Activation-engineering work uses direction vectors (mean-difference of contrast pairs), never single-coordinate edits.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat does mechanistic interpretability reveal about how concepts ARE encoded?
Sign in or go Pro to see the approach for this follow-up.
QIf dimensions aren't features, why does dimensionality reduction with PCA or UMAP work at all?
Sign in or go Pro to see the approach for this follow-up.
QCould you train an embedding model that DOES have interpretable dimensions?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Trying to "increase the sentiment" of an embedding by adjusting one coordinate. That coordinate doesn't represent sentiment. Semantic features are spread across the entire vector.