← All tutorials

Image-Text Contrastive Learning (CLIP)

On this page

1 Web captions are the labels 2 Two encoders share one embedding space 3 The diagonal holds the true pairs 4 Zero-shot classification asks which caption fits 5 CLIP in practice 6 References

In Contrastive Learning, each view picks its positive, the other view of the same image, out of a batch with InfoNCE. CLIP (Contrastive Language-Image Pre-training, Radford et al. 2021) uses the same loss between a photo and its caption. It trains an image encoder and a text encoder so that, every training step, each photo in the batch picks its own caption and each caption its own photo, using captions people already wrote on the web. The two trained encoders then classify new photos by caption, with no extra training.

Trainingphotocaptionimageencodertextencodertrue pairs onthe diagonalZero-shotclasspromptsnewphototextencoderimageencoderbest matchis the class
Figure 1. Top, training; bottom, classifying a new photo by comparing it with one caption per class.

1 Web captions are the labels

The web already pairs photos with text, so asking which caption goes with which photo needs no annotator.

Data400 million (image, text) pairs, called WIT (WebImageText)
Gathered by500,000 search queries, up to 20,000 pairs each
Image encodera ResNet (a convolutional network) or a Vision Transformer (ViT)
Text encodera Transformer
Batch32,768 pairs
Images to equal zero-shot accuracy predict the caption word by word 12× predict the caption’s words as a bag 4× pick the whole caption (CLIP) 1×
Figure 2. How CLIP was trained (sec. 2.2 to 2.5); bars from CLIP Fig. 2, where zero-shot means no training on the task (Section 4).

2 Two encoders share one embedding space

The image encoder and the text encoder run separately, and their outputs meet only when they are compared.

photoimage encoderResNet or ViTlinear projectionlength 1, gives Icaptiontext encoderTransformerlinear projectionlength 1, gives Tone dot product, the score
Figure 3. The two encoders (towers) never read each other; they meet only at the score on the right.

Each encoder's output goes through a linear projection into one shared space and is scaled to length 1. Nothing crosses between the two encoders, unlike Cross-Attention, where one stream reads the other; that is what lets CLIP encode each caption once and reuse it. The Q-Former's image-text contrastive stage keeps its queries and text apart the same way.

3 The diagonal holds the true pairs

CLIP trains each photo to pick its own caption out of the batch, and each caption to pick its own photo. A batch holds N photos and their captions. Their scores form an N by N table, rows for photos and columns for captions. The true pairs sit on the diagonal and the other N²−N pairings are wrong. Each row is cross entropy with the true caption as the class.

Score every photo against every caption, then make each row pick its true caption and each column its true photo; the loss averages both.
si,j = Ii · Tjτ
si,j scores photo i against caption j. I is the image encoder’s output. T is the text encoder’s output. Each passes through a linear projection into one shared space and is scaled to length 1, so their dot product · is a cosine between −1 and 1. The temperature τ divides it. Cosines alone span too narrow a range for a confident softmax; a small τ stretches the gaps, so the true caption can win clearly and the hardest wrong captions get most of the push. A large τ flattens every share toward 1/N.
ℓiimg = −log exp(si,i)Σj exp(si,j)
Photo i picks its caption out of all N. exp raises e (about 2.718) to each score, so every term is positive; the top is its own caption’s term, and Σj adds the terms of every caption in row i. The fraction is the photo’s softmax share for its true caption, and minus its log is near 0 when that caption wins and grows as it loses. This is InfoNCE with the row’s N captions as the candidates.
ℓjtxt = −log exp(sj,j)Σi exp(si,j)
Caption j picks its photo out of all N the same way, summing down column j instead of along a row. The softmax now runs over photos, so it penalizes a caption that scores high with the wrong photo.
L = 12N Σi=1N (ℓiimg + ℓitxt)
The loss L averages the 2N terms, one per photo and one per caption; the paper calls it a symmetric cross entropy. Rows catch a photo that picks the wrong caption, and columns catch a caption that picks the wrong photo. One wrong caption can win a photo’s row while that caption’s own column still picks its photo, so the two directions catch different mistakes and both encoders are corrected from both sides.

Figure 4. Six toy pairs; the dashed diagonal darkens as training runs.

3.1 The temperature is learned

CLIP learns τ as one more weight, updated by every gradient step with the encoders. It starts at 0.07, so 1/τ, a score multiplier, starts at about 14.3; the multiplier is capped at 100 (τ = 0.01) because larger ones made training unstable. Knowledge Distillation uses the same temperature in the opposite direction, raising it to soften a teacher’s output probabilities.

The snippet defines the loss L as clip_loss and calls it on toy features.

import torch
import torch.nn.functional as F

def clip_loss(img_feat, txt_feat, proj_img, proj_txt, log_scale):
    I = F.normalize(img_feat @ proj_img, dim=1)   # [N, D]
    T = F.normalize(txt_feat @ proj_txt, dim=1)
    scale = log_scale.exp().clamp(max=100.0)      # 1/τ, capped
    s = scale * I @ T.T                           # [N, N]
    diag = torch.arange(len(s))                   # true pairs
    l_img = F.cross_entropy(s, diag)
    l_txt = F.cross_entropy(s.T, diag)
    return (l_img + l_txt) / 2

N, D_IMG, D_TXT, D = 8, 2048, 512, 64
img_feat, txt_feat = torch.randn(N, D_IMG), torch.randn(N, D_TXT)
proj_img = torch.randn(D_IMG, D, requires_grad=True)
proj_txt = torch.randn(D_TXT, D, requires_grad=True)
log_scale = torch.tensor(1 / 0.07).log().requires_grad_()
L = clip_loss(img_feat, txt_feat, proj_img, proj_txt, log_scale)
  1. 1Projects and scales to length 1, giving I and T of Section 3 eq line 1.
  2. 2Scores si,j with the capped 1/τ, Section 3 eq line 1.
  3. 3Each photo picks its caption, Section 3 eq line 2.
  4. 4Each caption picks its photo, Section 3 eq line 3.
  5. 5Averages both directions, Section 3 eq line 4.
  6. 6Toy inputs for Section 3 eq line 1: features, projections and a learnable log of 1/τ.

4 Zero-shot classification asks which caption fits

To classify a photo with no training on the task, zero-shot, write each class as a caption and ask which caption the photo matches best.

new photoa photo of a dog.a photo of a cat.a photo of a car.a photo of a tree.
Figure 5. Toy shares over four prompts; the dog prompt wins.

The snippet uses OpenAI’s clip package to classify one photo with four caption prompts.

import clip
import torch
import torch.nn.functional as F
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("RN50", device=device)
names = ["dog", "cat", "car", "tree"]
prompts = [f"a photo of a {c}." for c in names]
tokens = clip.tokenize(prompts).to(device)
photo = Image.new("RGB", (224, 224))       # toy photo
x = preprocess(photo).unsqueeze(0).to(device)
with torch.no_grad():
    W = F.normalize(model.encode_text(tokens), dim=-1)  # [4, D]
    h = F.normalize(model.encode_image(x), dim=-1)      # [1, D]
pred = names[(h @ W.T).argmax()]
  1. 1Loads a trained CLIP, the two encoders of Figure 3.
  2. 2Class names become caption prompts, the text side of Figure 5.
  3. 3One preprocessed photo, the new photo in Figure 5.
  4. 4Text rows W, encoded once, and photo embedding h with no gradients, Section 3 eq line 1 without τ, since the multiplier 1/τ cannot change the winner.
  5. 5The best-matching caption is the class, the winning bar in Figure 5.

The class matrix W works like the weights of a linear probe (one linear layer trained on frozen features) with no bias, written by the text encoder from class names instead of learned from labeled photos (sec. 3.1.2). The paper’s main model, ViT-L/14@336px, is a large Vision Transformer that cuts a 336-pixel photo into 14-pixel patches.

ImageNet top-1 (%)CLIP zero-shot, no ImageNet labels76.2original ResNet-50, 1.28 million labelsmatched050100
Figure 6. With no labels, CLIP ViT-L/14@336px matches a supervised ResNet-50 (sec. 3.1.3).
Gain from the prompt (points)“a photo of a {label}.” over the bare name+1.380 prompts averaged, over one+3.5012345
Figure 7. Prompt wording adds accuracy at no training cost (sec. 3.1.4).

5 CLIP in practice

One training step reuses clip_loss, the sizes and the parameters of the Section 3.1 snippet, with small stand-ins for the ResNet and the Transformer.

import torch
import torch.nn as nn

VOCAB, CTX = 49408, 76
image_tower = nn.Sequential(nn.Flatten(), nn.Linear(3 * 64 * 64, D_IMG))
text_tower = nn.EmbeddingBag(VOCAB, D_TXT)      # stand-in Transformer
towers = [*image_tower.parameters(), *text_tower.parameters(), proj_img, proj_txt]
decay = [p for p in towers if p.ndim >= 2]
no_decay = [p for p in towers if p.ndim < 2] + [log_scale]
groups = [dict(params=decay, weight_decay=0.2), dict(params=no_decay, weight_decay=0.0)]
opt = torch.optim.AdamW(groups, lr=5e-4)

photos = torch.rand(N, 3, 64, 64)               # one batch of pairs
tokens = torch.randint(0, VOCAB, (N, CTX))
L = clip_loss(image_tower(photos), text_tower(tokens), proj_img, proj_txt, log_scale)
opt.zero_grad()
L.backward()
opt.step()
  1. 1Stand-in image and text encoders, the two towers of Figure 3.
  2. 2AdamW with weight decay 0.2 on matrices and 0 on biases and gains (sec. 2.5) and on log(1/τ), the τ of Section 3 eq line 1, as open_clip does.
  3. 3One batch of pairs through clip_loss, Section 3 eq lines 1 to 4.
  4. 4One update of the encoders, both projections and τ together.
Actsevery optimizer step; τ is updated with the weights
Plugs inpretraining only; then the encoders stay frozen for zero-shot or a probe
Keepsthe two encoders, the two projections and log(1/τ)
Settings32 epochs, Adam with weight decay 0.2 on all weights that are not gains or biases, learning rate 5×10−4 for RN50, 2000 warm-up steps (sec. 2.5, Tables 18 and 19)
Cost18 days on 592 V100 GPUs for the largest ResNet, 12 days on 256 for the largest ViT (sec. 2.5)
Codeopenai/CLIP (released models), mlfoundations/open_clip (training)
Gotchaswithout the cap on 1/τ training goes unstable; no weight decay on gains and biases (sec. 2.5); open_clip also skips the temperature, since decay drags 1/τ toward 1; each GPU holds part of the batch, so embeddings are gathered from all GPUs to score every pair; the paper needed mixed precision (16-bit arithmetic) and gradient checkpointing (recomputing activations instead of storing them) to fit
Knob, defaultRaise itLower it
Start of τ in si,j; 0.07, then learneda softer first guess; training moves τ anywaya sharper first guess; training moves τ anyway
Cap on 1/τ; 100, so τ stays at 0.01 or moreshares may grow sharper; the cap exists to stop training going unstablescores stay soft for every pair
N, the batch; 32,768more wrong captions per row and column, more memoryan easier choice per row; at N = 1 there is no wrong caption to reject; the paper gives no batch ablation

Non-Contrastive Self-Supervised Learning comes next, where the negatives go away entirely.

6 References