What does a contrastive loss optimize?

A **contrastive loss** is a training objective designed around pairs of inputs. Each batch (a group of training examples) has positive pairs (related, like a question and its answer) and negative pairs (unrelated, like the question and a random document). The loss pulls positives together in vector space and pushes negatives apart. This is what creates the geometry every embedding application depends on. Before training, the embedding space is random. After enough contrastive updates, similar inputs land near each other and dissimilar inputs land far apart, automatically, without any human annotating dimensions. Three variants dominate in embedding training. **InfoNCE** is cross-entropy over a softmax (scores turned into probabilities) of pairwise similarities. The positive pair is the "correct class" among N candidates. **Triplet loss** uses a hard margin: anchor-positive distance must be smaller than anchor-negative distance by some margin. MNRL (Multiple Negatives Ranking Loss) is a practical variant of InfoNCE that treats every other example in the batch as a negative: no explicit negative mining required. The family won because it's data-efficient (no token labels needed, just pairs). Its geometry matches the similarity search use case, and it scales to billions of pairs.

**Contrastive training optimises a pull and push objective: positive pairs (semantically related inputs) are pulled closer in vector space. Negative pairs (unrelated inputs) are pushed apart.** The cumulative effect across millions of pairs is a geometry where cosine similarity tracks semantic similarity. The dominant formulation is **InfoNCE**: cross-entropy over a softmax of similarities, treating the positive pair as the correct class among N candidates. The loss is: $$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, k^+) / \tau)}{\sum_i \exp(\text{sim}(q, k_i) / \tau)}$$ where `sim` is typically the cosine similarity, `k^+` is the positive, `k_i` ranges over positive plus negatives, and `τ` is a temperature. Two related formulations show up regularly. **Triplet loss** uses a margin-based hinge (anchor-positive closer than anchor-negative by margin `m`). MNRL treats every other batch example as an in-batch negative: fast and practical, no explicit mining. What the loss buys is twofold. The geometry is **angular** because the loss operates on normalised dot products: magnitudes don't matter, only directions. And the geometry is task-aligned: the model arranges the space to make the contrastive loss as low as possible. This directly produces the property downstream applications need.