Explain how the InfoNCE loss shapes embedding geometry during training

Attempt it

Walk through what the InfoNCE loss optimizes, and why models trained this way 'live' in angular space rather than Euclidean space.

Free · 2 AI evals / day

InfoNCE shapes the embedding space by repeatedly running a multiple-choice task. Each training step, the model sees a query and N candidate documents: one positive (the right answer) and N-1 negatives. The model has to assign similarity scores to all N candidates, then softmax those scores into a probability distribution. Cross-entropy penalises the model for putting low probability on the positive. The similarity score is **cosine similarity**: the dot product of L2-normalised vectors. Because vectors are normalised to unit length before the dot product, only the **direction** of each vector matters; the magnitude is stripped away. This is why the geometry the model learns is **angular**. Gradients push the query and positive vectors to point in the same direction (cosine close to 1, angle close to 0). Gradients push the query and negatives toward perpendicular (cosine close to 0, angle close to π/2). The model isn't learning Euclidean distances; it's learning directions on the unit sphere. This matches the inference-time metric. Production retrieval uses cosine similarity to find nearest neighbours. The training objective and the inference metric agree, which is why InfoNCE-trained embeddings work so well for similarity search.

**InfoNCE treats each positive pair as a 1-of-N classification problem and applies cross-entropy over softmax-normalised similarities.** For a query `q`, one positive `k+`, and N-1 negatives `k-_i`: $$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(q \cdot k^+ / \tau)}{\sum_i \exp(q \cdot k_i / \tau)}$$ The loss is minimised by making `q · k+` large relative to every `q · k-_i`. The gradient pulls the query and positive toward each other and pushes the query away from every negative. The critical detail is that `q` and `k_i` are typically **L2-normalised before the dot product**. With unit-length vectors, the dot product equals cosine similarity, which depends only on the angle between vectors. Magnitudes are irrelevant. This forces the model into **angular geometry**: positives end up at small angles to their queries, negatives at angles near π/2 or beyond. The consequence for inference is direct. Cosine similarity is the natural retrieval metric because it's the metric the model was trained on. If you queried with Euclidean distance on the same vectors, you'd get similar but slightly degraded results: the geometries are related but the training only optimised the angular component. Negatives come from two main sources. **In-batch negatives** treat every other example in the batch as a negative: free, fast, but become trivial after early training. Mined hard negatives (BM25-based or prior-model top-K) keep the loss informative by including semantically similar non-positives. Production embedding models rely heavily on the latter.

Quick reference

TL;DR

InfoNCE optimises cross-entropy over softmax-normalised cosine similarities, sculpting an angular geometry where positives sit near angle zero and negatives near orthogonal.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture every sentence as an arrow pointing somewhere on a giant sphere. The training game is to make arrows for related sentences point in the same direction and arrows for unrelated sentences point in different directions. The length of each arrow doesn't matter, only where it points. After enough rounds, the sphere becomes organised: travel guides cluster on one patch, recipes on another, programming tutorials on a third. To find related sentences at query time, you just check which arrows point the same way as the query's arrow.

Key concepts

Embeddings

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Write the InfoNCE formula. Identify the normalisation step. Explain why this produces angular geometry. Walk through the gradient (pull positives, push negatives). Connect to cosine as the inference metric. Close with the role of negatives and the temperature.

Real-world examples

Real products, models, and research that use this idea.

OpenAI text-embedding-3 series trains with InfoNCE and ships normalised vectors so cosine and dot product are interchangeable.
BGE-M3 applies InfoNCE with mined hard negatives across multilingual training, producing 1024-dim normalised embeddings.
CLIP uses symmetric InfoNCE on image-text pairs, sharing the same angular-geometry property.
Voyage v3 and Cohere embed-v4 are both InfoNCE-trained and emit L2-normalised outputs by default.
Sentence-Transformers MNRL is InfoNCE with in-batch negatives: the default for fine-tuning custom embeddings.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat happens if you train with cosine but query with raw dot product on non-normalised vectors?

Sign in or go Pro to see the approach for this follow-up.

QWhy does temperature τ have such a strong effect on training dynamics?

Sign in or go Pro to see the approach for this follow-up.

QWhat does the gradient look like when most negatives are trivially far from the query?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Describing InfoNCE as a Euclidean-distance loss. The loss operates on cosine of normalised vectors: magnitudes are stripped out, so the geometry is angular, not distance-based.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.