Critique this claim about CLIP using cross-attention between towers

**The error is the entire claim about cross-attention.** CLIP does not use cross-attention between its image and text towers. It is a **dual-encoder** (two-tower) model: a vision transformer processes images, a text transformer processes text, and the two encoders never see each other's tokens during their forward passes. Each tower produces one pooled embedding per input. Alignment happens at the **loss function**, not at attention. Both embeddings are linearly projected into a shared d-dimensional space. The training loss is **contrastive** (InfoNCE-style): for a batch of N image-text pairs, the model is rewarded when each image's embedding has high cosine similarity with its matched caption's embedding and low similarity with all other captions in the batch. Why this matters at inference: - Embed an image once, store its 512- or 768-dim vector. - Embed an arbitrary text query once. - Compute one dot product per candidate, ranked by cosine similarity. This is what makes CLIP fast at billion-scale retrieval. If the towers were coupled by cross-attention, every (image, text) pair would need a fresh joint forward pass, and a 1B-image catalog would be unscoreable. The models that DO use cross-attention between vision and text are BLIP-2, Flamingo, LLaVA, and other vision-language MODELS for captioning and VQA. They trade retrieval efficiency for richer joint reasoning.

**The error is the architecture claim.** CLIP is a **dual-encoder contrastive model**, not a cross-attention model. The vision transformer and the text transformer run independent forward passes, never share parameters, and never attend to each other's tokens. Each tower outputs one pooled embedding. Alignment is produced by the **InfoNCE loss** over those embeddings: project both into a shared d-dim space, compute pairwise cosine similarities for a batch of N image-text pairs, and apply softmax cross-entropy that pulls matched pairs together and pushes mismatched pairs apart: $$\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(\text{sim}(z_i^I, z_i^T) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(z_i^I, z_j^T) / \tau)}$$ The retrieval property follows: embed an image once, embed a text query once, compare with a dot product. That is why CLIP scales to billion-image catalogs. **Cross-attention is the wrong technique for CLIP's goal.** Models that DO use cross-attention between modalities (BLIP-2, Flamingo, LLaVA) target captioning, VQA, and joint reasoning, where every (image, text) pair really does need a coupled forward pass. They trade retrieval scalability for richer interaction. The interview tell is: if someone says "CLIP uses cross-attention" they are confusing it with a downstream VLM, not the original 2021 CLIP recipe.