What does CLIP stand for and what is its key contribution?
CLIP trains a vision encoder and a text encoder jointly on 400M image-caption pairs so their outputs land in a shared vector space.
Imagine two students learning the same secret code from opposite ends. One only sees pictures, the other only reads captions. Every day they get matched pairs (picture, caption) and a pile of mismatched ones. Their job is to score the matches high and the mismatches low. After enough rounds, both students agree on the same numbers for the same idea, even though one looked at pixels and the other at words. That shared map is what CLIP produces. You can drop a picture of a dog onto it, drop the words a dog onto it, and they land in the same spot. Now you can search across pictures with words, or label pictures by writing what they show.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
CLIP is one of the most-cited models of the last five years, and for good reason: it is the cleanest demonstration of how to align two modalities through a contrastive loss instead of through architecture. Before CLIP, every cross-modal task (image captioning, visual question answering, image search by text) required either a dedicated supervised dataset or a hand-engineered fusion module. After CLIP, the recipe was uniform: encode each modality independently, pull matched pairs together, push mismatched pairs apart, and rely on the loss to do the alignment.
This deep dive unpacks the recipe, explains why the shared embedding space falls out of the objective, and traces the family tree from CLIP through SigLIP and EVA-CLIP to the 2026 multimodal-embedding landscape.
The acronym and what it commits to
CLIP stands for Contrastive Language-Image Pretraining. Each word maps to a concrete design choice.
Contrastive: the training signal is not classification, regression, or generation. It is a pairwise comparison. For each training step, the model sees a batch of matched (image, caption) pairs and is rewarded for assigning higher similarity to matched pairs than to mismatched pairs in the same batch. This is the InfoNCE objective.
Language-Image: the two modalities being aligned. CLIP uses a transformer text encoder and a vision encoder (a ViT in the larger checkpoints). The two encoders never share weights.
Pretraining: the model is trained once at scale, with no task-specific labels. Every downstream use (zero-shot classification, retrieval, conditioning a diffusion model) is a deployment of the same frozen encoders. There is no fine-tuning step in the original CLIP recipe, which is part of what made the release so striking.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI CLIP ViT-L/14: the original 2021 release that established the recipe and is still cited as the baseline.
- Google SigLIP-2: uses a sigmoid contrastive loss that scales past 30k batch sizes, currently a top open vision-text encoder in 2026.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is batch size so important during CLIP training, and what did SigLIP change about it?
InfoNCE normalizes the loss across the batch, so every example competes against every other example as a negative. Bigger batch means harder negatives and stronger contrast. SigLIP's sigmoid loss treats each pair independently and removes that batch-wide softmax, which lets it scale to far larger effective batches.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling CLIP a generator or a classifier. CLIP is two encoders that produce embeddings; image generation comes from diffusion models, and zero-shot classification is a downstream USE of CLIP, not what CLIP outputs.
60 second bullets to scan on the way to the call.
Spell out the acronym (Contrastive Language-Image Pretraining)
Two-encoder architecture (vision encoder + text encoder)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.