How does CLIP-style training produce a joint image-text embedding space?

**CLIP** trains two encoders side by side. A vision encoder turns images into vectors. A text encoder turns captions into vectors. They have separate weights and never see inside each other. The training data is a large pile of (image, caption) pairs scraped from the web: about 400 million of them in the original CLIP. Each training step takes a batch of pairs, say 32k of them. The model embeds every image and every caption, then computes the cosine similarity between every image-caption combination in the batch. The loss is a contrastive objective. For each image, the cosine to its OWN caption should be high, and the cosine to every OTHER caption in the batch should be low. The same rule runs in reverse for each caption against the images. This is called symmetric InfoNCE: two sides of the same loss, one for image to text and one for text to image. Over many batches, the only way to keep that loss low is for the two encoders to produce vectors in the same coordinate system. An image of a dog and the text 'a dog' end up in nearly the same place. That shared coordinate system is the joint embedding space. After training, you can embed text and images independently and compare them with cosine similarity.

**CLIP** trains a vision encoder and a text encoder in parallel on roughly 400M (image, caption) pairs. The two encoders never share weights. They share only their output dimension. The training step is elegant. For each batch of N pairs, compute the N x N cosine-similarity matrix between every image and every caption. The diagonal contains the matched pairs (image_i with text_i). Everything off-diagonal is a mismatched pair and treated as a negative. The symmetric InfoNCE loss maximizes the diagonal and minimizes the off-diagonal: $$\mathcal{L} = -\frac{1}{2N} \sum_i \left[ \log \frac{e^{S_{ii}/\tau}}{\sum_j e^{S_{ij}/\tau}} + \log \frac{e^{S_{ii}/\tau}}{\sum_j e^{S_{ji}/\tau}} \right]$$ The two terms are the image to text and text to image directions. The temperature `\tau` is a learnable scalar that controls how peaked the softmax is. The shared space falls out of this. Since matched pairs must be close and mismatched pairs must be far, the only stable equilibrium across hundreds of millions of pairs is one where the two encoders agree on the geometry of the output space. An image of a dog and the text 'a dog' end up in the same region, not because the encoders look alike, but because any other arrangement increases the loss. Deployment is asymmetric and that's part of the recipe's win. The image encoder runs once per image (you pre-embed the corpus). The text encoder runs per query (it's smaller and cheaper). Modern variants (SigLIP, OpenCLIP, EVA-CLIP) change the loss or the scale but keep the two-tower, shared-space pattern. Downstream uses: text to image search, zero-shot classification by embedding label strings, and multimodal RAG.