Explain how the InfoNCE loss shapes embedding geometry during training

InfoNCE shapes the embedding space by repeatedly running a multiple-choice task. Each training step, the model sees a query and N candidate documents: one positive (the right answer) and N-1 negatives. The model has to assign similarity scores to all N candidates, then softmax those scores into a probability distribution. Cross-entropy penalises the model for putting low probability on the positive. The similarity score is **cosine similarity**: the dot product of L2-normalised vectors. Because vectors are normalised to unit length before the dot product, only the **direction** of each vector matters; the magnitude is stripped away. This is why the geometry the model learns is **angular**. Gradients push the query and positive vectors to point in the same direction (cosine close to 1, angle close to 0). Gradients push the query and negatives toward perpendicular (cosine close to 0, angle close to π/2). The model isn't learning Euclidean distances; it's learning directions on the unit sphere. This matches the inference-time metric. Production retrieval uses cosine similarity to find nearest neighbours. The training objective and the inference metric agree, which is why InfoNCE-trained embeddings work so well for similarity search.

**InfoNCE treats each positive pair as a 1-of-N classification problem and applies cross-entropy over softmax-normalised similarities.** For a query `q`, one positive `k+`, and N-1 negatives `k-_i`: $$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(q \cdot k^+ / \tau)}{\sum_i \exp(q \cdot k_i / \tau)}$$ The loss is minimised by making `q · k+` large relative to every `q · k-_i`. The gradient pulls the query and positive toward each other and pushes the query away from every negative. The critical detail is that `q` and `k_i` are typically **L2-normalised before the dot product**. With unit-length vectors, the dot product equals cosine similarity, which depends only on the angle between vectors. Magnitudes are irrelevant. This forces the model into **angular geometry**: positives end up at small angles to their queries, negatives at angles near π/2 or beyond. The consequence for inference is direct. Cosine similarity is the natural retrieval metric because it's the metric the model was trained on. If you queried with Euclidean distance on the same vectors, you'd get similar but slightly degraded results: the geometries are related but the training only optimised the angular component. Negatives come from two main sources. **In-batch negatives** treat every other example in the batch as a negative: free, fast, but become trivial after early training. Mined hard negatives (BM25-based or prior-model top-K) keep the loss informative by including semantically similar non-positives. Production embedding models rely heavily on the latter.