Describe how to build an image search system where users query with free form text. Explain which embeddings you use, how the query and images are compared, and why this works without any per image labels or captions.
Embed every image offline with CLIP's image encoder, embed the text query with its text encoder, and rank by cosine similarity — the shared space replaces labels with learned cross-modal alignment.
Imagine a library where every book and every topic card is placed on one giant map by meaning, not by shelf number. To find books about volcanoes, you drop your topic card on the map and grab whatever sits nearest — no one had to label each book first. Text to image search works the same way. CLIP put pictures and sentences on one shared map during training. So your typed query lands somewhere, and the closest pictures are your results, even though nobody tagged those pictures by hand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Text to image search sounds like it should require a mountain of labeled data: someone tagging each photo with keywords so a text query can match. The interesting part of this question is that the modern answer skips labeling entirely, and understanding why is what separates a rote answer from a strong one.
The enabling idea is a shared embedding space. If a sentence and a picture can be turned into vectors that live in the same geometry, then comparing them is just measuring distance. CLIP produces exactly that space, which turns search into a nearest-neighbor lookup.
This deep dive walks through the two-phase architecture, the math of ranking, the production details that make or break the system, and the limits you should call out before an interviewer does.
The shared space is the whole trick
CLIP trains an image encoder and a text encoder together with a contrastive loss. The loss pulls each image toward its true caption and pushes it away from every other caption in the batch. After training, a picture and a sentence describing it land near each other in one space, and unrelated pairs land far apart.
That single fact is what makes label-free search possible. In a classic system, you would tag each image with keywords and match query terms against those tags. Adding a new searchable concept meant someone re-tagging the corpus. Here, the tags are implicit: the image vector already encodes meaning in a form the text vector can be compared to, and any new query concept just becomes a new point in the same space.
So the design question "how does it work without labels?" has a one-line answer. Contrastive pretraining did the alignment work that manual labeling used to do. A query vector can be scored against image vectors that no human ever annotated, because both vectors were placed in the same geometry by the same training objective.
It helps to picture the geometry. Every image in your library is a point on a high-dimensional sphere, since CLIP normalizes its outputs. A text query is another point on that same sphere. "Nearest images" means "smallest angle to the query," and the angle is meaningful precisely because the contrastive loss spent training making true image-caption angles small. Nothing about this requires the images to have ever been described in words by a person.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import torch, faiss, clip
model, preprocess = clip.load("ViT-B/32")
# Offline: embed and index every image once
img_vecs = torch.cat([model.encode_image(preprocess(im).unsqueeze(0)) for im in images])
img_vecs = torch.nn.functional.normalize(img_vecs, dim=-1)
index = faiss.IndexFlatIP(img_vecs.shape[1]) # inner product on unit vecs = cosine
index.add(img_vecs.numpy())
# Online: embed the text query and search
q = model.encode_text(clip.tokenize(["a red bicycle"]))
q = torch.nn.functional.normalize(q, dim=-1)
scores, ids = index.search(q.numpy(), k=10)Real products, models, and research that use this idea.
- OpenAI CLIP backing a text to image search over a photo library with a FAISS index.
- pgvector or Pinecone storing CLIP image embeddings for production semantic image search.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you scale this to a billion images while keeping query latency low?
Discuss ANN indexes like HNSW or IVF-PQ, vector quantization for memory, and the recall versus latency tradeoff.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Re-embedding the whole image library at query time. Images are embedded once offline and indexed; only the text query is embedded per search.
60 second bullets to scan on the way to the call.
Which model family supplies the two aligned encoders
What gets embedded offline versus per query
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.