SigLIP versus CLIP: how a sigmoid loss changes vision encoder pretraining

Flashcard·Easy·4.0 · 0·~30s·Asked atBytedance Hcl Tesla

Attempt it

CLIP uses a softmax contrastive loss computed over the whole batch, so every image is scored against every text in the batch and quality depends on very large batch sizes. SigLIP replaces that with a pairwise sigmoid loss: each image-text pair is scored independently as match or no match. That removes the global softmax normalization, trains stably at smaller batch sizes, and scales more efficiently, which is why SigLIP-family encoders became a default pick for LLaVA-style VLM connectors and document-heavy vision stacks.

Your answer

**CLIP** and **SigLIP** are both contrastive image text encoders, and they look almost identical from the outside: an image tower and a text tower trained to produce embeddings where matched pairs are close and mismatched pairs are far. The difference is in the loss. CLIP uses a softmax over the whole batch. For each image, the model must rank the correct caption above every other caption in the batch. That global normalisation means a small batch contains few negatives, and the model learns weak signal. To train CLIP well you need batches in the tens of thousands of samples, which is expensive. SigLIP replaces that with a sigmoid loss per pair. Each image-caption pair gets a yes-or-no score independently of the others. No global normalisation. The model can train stably at batch sizes of 1k to 4k that fit on smaller hardware, and it scales smoothly when you do give it more compute. VLM builders care because the encoder choice ripples downstream. SigLIP-family encoders give you a strong starting point for the vision tower without requiring CLIP-scale infrastructure, which is why LLaVA-NeXT, Qwen2-VL, and many open-source VLMs use SigLIP variants by default.

**SigLIP** swaps **CLIP**'s softmax contrastive loss for a pairwise sigmoid loss. In CLIP, for a batch of N images and N captions, you compute the N by N similarity matrix and apply a softmax row-wise and column-wise. Every image competes with every other image as a hard negative. The loss for a single positive pair depends on the entire batch through the normalisation. That global softmax is what forces CLIP to train at very large batch sizes; with only 256 negatives, the contrastive signal is too easy and the model under-trains. SigLIP changes the loss to: $$\mathcal{L} = -\sum_{i,j} \log \sigma\big(z_{ij} (t \cdot x_i \cdot y_j + b)\big)$$ where `z_ij` is +1 for the diagonal (positive pair) and -1 elsewhere. Each pair is a binary classification, fully independent. No softmax, no global normalisation. **Three downstream wins:** (1) batch size becomes a tuning knob rather than a hard constraint, so smaller teams can train competitive encoders; (2) scaling laws stay clean when batches do grow, so you do not get diminishing returns from softmax saturation; (3) the loss is well behaved when you scale image and text towers asymmetrically. That is why SigLIP and SigLIP 2 are the default vision tower in modern open-source VLMs like LLaVA-NeXT and Qwen2-VL, and the connector training is more stable on them than on CLIP equivalents.

TL;DR

CLIP normalises matches across the whole batch with a softmax; SigLIP scores each image-text pair independently with a sigmoid, training stably at smaller batches and scaling more cleanly.

Memory aid

Easy to grasp

Think of pairing photos with captions at a party. CLIP plays a competition: every photo is held up next to every caption in the room and the loudest match wins; if the room is small, the competition is too easy to learn anything. SigLIP plays a simpler game: for each photo and caption pair you only answer yes or no, is this a match. You no longer need the whole room; you only need each pair. That smaller demand means SigLIP trains well on smaller machines and scales up cleanly, which is why most modern open-source vision language models use SigLIP-family encoders to connect images to the language model.

Key concepts

Multimodal AI

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

When a VLM builder picks a vision encoder, they are choosing the eye that the language model will look through. The choice has been dominated for years by CLIP and its open-source descendants (OpenCLIP, EVA-CLIP). In 2023 SigLIP arrived with a deceptively small change: replace the contrastive softmax with a pairwise sigmoid. The mathematical change is one line; the operational consequences reshape how vision encoders are trained and which ones end up shipping in production VLMs.

This walkthrough goes through the loss math, the practical batch-size and scaling implications, and the downstream choice every modern VLM builder is now making about which encoder to start from.

Mental model: CLIP plays a tournament where every pair competes against every other pair, demanding a huge room of contestants. SigLIP plays solo: each pair answers yes or no on its own, and the room can be any size.

The loss change in math

CLIP trains with an InfoNCE-style softmax contrastive loss. For a batch of N image-caption pairs, the model computes the N by N similarity matrix S where S_ij = sim(image_i, caption_j). The loss applies softmax along each row and column and treats the diagonal as the positive class:

\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_i \left[ \log \frac{e^{S_{ii}/\tau}}{\sum_j e^{S_{ij}/\tau}} + \log \frac{e^{S_{ii}/\tau}}{\sum_k e^{S_{ki}/\tau}} \right]

The denominator is the partition function over the entire batch. Every positive pair's loss depends on every other pair in the batch through that normalisation.

SigLIP replaces this with a pairwise sigmoid loss:

\mathcal{L}_{\text{SigLIP}} = -\frac{1}{N} \sum_{i, j} \log \sigma\!\left(z_{ij} (t \cdot S_{ij} + b)\right)

where z_ij = +1 on the diagonal and -1 elsewhere, and t, b are learnable temperature and bias scalars. Each (i, j) pair is now an independent binary classification. The full matrix still has N squared entries but each contribution is decoupled.

That single change removes the global normalisation and lets the loss decompose cleanly across devices and across batch chunks.

Why batch size matters less under SigLIP

Downstream consequences for VLM builders

When to choose what

Production realities and 2026 model lineup

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

# SigLIP vs CLIP loss in pseudocode
import torch
import torch.nn.functional as F

def clip_loss(image_emb, text_emb, temperature=0.07):
    """InfoNCE softmax over the batch."""
    logits = image_emb @ text_emb.T / temperature
    targets = torch.arange(logits.size(0), device=logits.device)
    # Symmetric: image-to-text + text-to-image
    return 0.5 * (F.cross_entropy(logits, targets) + F.cross_entropy(logits.T, targets))

def siglip_loss(image_emb, text_emb, t, b):
    """Pairwise sigmoid; t and b are learnable scalars."""
    logits = t * (image_emb @ text_emb.T) + b
    N = logits.size(0)
    labels = 2 * torch.eye(N, device=logits.device) - 1  # +1 diag, -1 elsewhere
    return -F.logsigmoid(labels * logits).mean()

Aspect	CLIP	SigLIP
Loss	Softmax (InfoNCE) over batch	Per-pair sigmoid
Normalisation	Global across batch	None
Batch size	32k+ for clean signal	Stable from 4k upward
Scaling law	Plateaus past a regime	Monotonic up to 32k
Default in 2026 VLMs	Retrieval baselines	LLaVA-NeXT, Qwen2-VL, PaliGemma

Real products, models, and research that use this idea.

LLaVA-NeXT uses SigLIP as its vision tower for stronger document and chart understanding than CLIP equivalents.
Qwen2-VL builds on a SigLIP-family encoder with naive dynamic resolution and outperforms CLIP-towered VLMs on OCR benchmarks.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the learnable bias `b` matter so much in practice?

It encodes the prior over positive pairs in the dataset and prevents the early-training regime from collapsing to predict-negative-for-everything; without it, sigmoid loss is harder to stabilise.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Thinking SigLIP changes the architecture. The encoder shape is roughly the same; only the loss changes, and that change is enough to alter the entire compute and batch-size profile.

60 second bullets to scan on the way to the call.

The mathematical difference between InfoNCE softmax and pairwise sigmoid loss
Why CLIP needs very large batch sizes

Primary sources. Browse if you want the original framing.