CLIP jointly trains an image encoder and a text encoder so matching image-caption pairs land close in one shared space — unlocking zero-shot classification and search without labels.
Imagine a party where every photo is handed a name tag describing it. CLIP learns to seat each photo next to the right description and far from the wrong ones, all in the same room. After enough practice, any new photo drifts toward the words that fit it best. That shared room is the trick. Because pictures and sentences sit in the same place, you can ask "which photo matches this sentence?" just by checking who sits nearby — no one ever had to label the photos by hand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Before CLIP, image classifiers were trained on fixed label sets like the thousand ImageNet classes. Adding a new class meant collecting labeled examples and retraining. CLIP broke that loop by learning from a different kind of supervision: the captions that already accompany images across the web. No one hand-labeled those pairs into categories — the caption itself is the supervision.
The shift matters because it changes what the model can do at test time. Instead of a frozen list of classes, you describe the classes in words at inference. "A photo of a tabby cat" becomes a query you can score against any image. That flexibility is why CLIP shows up under so many systems in 2026, from search to moderation to the vision tower of multimodal language models.
This deep dive walks through the dual-encoder setup, the exact contrastive loss, why the shared space makes both classification and retrieval fall out for free, and where the approach hits its ceiling.
Two towers that never see each other's internals
CLIP is built from two independent encoders. The image encoder is typically a Vision Transformer or a ResNet that maps an image to a single fixed-length vector. The text encoder is a Transformer that maps a caption to its own fixed-length vector. Both project into a space of the same dimension, and both outputs are normalized to unit length so that a dot product equals cosine similarity.
The key design point is that the two towers do not share weights. They are tuned only by the shared loss, which forces their outputs into a common geometry. Think of two translators who never speak the same language but are graded on whether their summaries of the same scene end up identical. Over millions of examples, the only way to keep scoring well is to converge on a matching internal vocabulary, even though neither tower ever inspects the other's weights.
At the end of training you can throw away whichever tower you do not need. Retrieval systems keep the image tower to index a library and the text tower to embed queries. Vision-language models like LLaVA keep only the image tower and feed its patch features into a language model. The towers are reusable parts, which is a large reason CLIP spread so fast.
It is worth being precise about the output. For retrieval and classification, CLIP uses the single pooled vector per image — one summary of the whole picture. When CLIP serves as a VLM's vision encoder, the system instead taps the per-patch features before pooling, because a language model wants many local tokens, not one global summary. Same tower, two ways to read its output.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI CLIP — the original dual-encoder powering zero-shot image classification and text to image search.
- LLaVA — uses a CLIP image encoder as its vision tower before projecting features into the LLM.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does CLIP train with very large batch sizes?
Connect batch size to the number of negative pairs available per contrastive step and how that sharpens alignment quality.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming CLIP "understands" an image in detail. Its contrastive objective is global and coarse — it matches whole image to whole caption, not fine details.
60 second bullets to scan on the way to the call.
The two towers CLIP trains and what each one outputs
What the contrastive loss rewards versus punishes
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.