Self-supervised learning (SSL) makes its labels from the data itself. It trains an encoder, the network that turns a photo into a vector, its embedding, on unlabeled data; this stage, pretraining, updates it every training step. A few labels then check it once, after pretraining: a probe trains one small layer on top of the frozen encoder, while fine-tuning retrains every weight.
Figure 1. The five pages that follow this one, and what each method does.
1 Labels are expensive, but unlabeled data has structure
Photos already repeat objects, parts and scenes, so comparing edited copies of them can teach those things without a label.
Figure 2. Labels arrive only at the end, to check or adapt the encoder.
Most methods compare views, copies made by an augmentation (a random crop, recolor or mask); views look different but show the same thing.
Figure 3. Three views of one drawing show different pixels and the same object.
2 Pretext tasks turn unlabeled data into questions with answers
A pretext task (Doersch et al. 2015) is a training question whose answer comes from the data itself, such as where a patch sat in its photo; answering it forces the encoder to learn objects, their parts and the scene.
Figure 4. Each question needs objects, their parts and the scene, not raw pixel values.
3 Three families of methods learn without labels
Two views of one photo are a positive pair; views of other images are negatives. If the encoder gave every input the same output, every pair would match and nothing would be learned; that failure is collapse. Contrastive learning stops it by pushing negatives away. Non-contrastive methods match a target that gets no gradient, often from a momentum encoder (a copy whose weights follow a moving average of the encoder’s). Masked image modeling has no collapse to stop, because it must rebuild the real hidden pixels.
Figure 5. At (1), some methods on later pages (MoCo, BYOL, DINO) use a momentum encoder.
4 A linear probe measures what a frozen encoder learned
A linear probe freezes the encoder (its weights stop changing) and trains one linear layer on top to predict labels.
The probe turns an image into class probabilities through the frozen encoder and one linear layer; only that layer is trained, by cross entropy on the labels.
h =f(x)
The frozen encoder f turns one labeled image x into its embedding h, the one pretraining learned. Because f never changes, h is the same every time the image comes back, so the probe sees exactly what pretraining produced and nothing more.
ŷ =softmax(Wh + b)
Each class gets one score. The weight matrix W holds one row per class, and a score is that row’s weighted sum of the numbers in h. The bias b adds one offset per class. Softmax turns the scores into probabilities ŷ (y-hat) that add up to 1, and only W and b train. The score is linear in h, so each class boundary is a straight cut through the embedding space. A high probe score therefore means the frozen encoder already put the classes on separable sides; a deeper head could do some of the sorting itself and hide how good the features are.
L =−1N Σn log ŷn[yn]
The cross entropy L is the loss the probe trains on. For each of the N labeled images, ŷn[yn] is the probability the probe gives that image’s true label yn; Σ adds up minus the log of that probability over the images, and 1/N takes the average. A confident right answer (probability near 1) adds almost 0 and a confident wrong one adds a lot, so L is low when the probe puts high probability on the true labels. Training lowers it by gradient descent: each step moves W and b a short way against the gradient, the direction in which L rises fastest, and the step size (lr in the code below) sets how far.
The probe trains once, after pretraining, on labeled data (the MoCo paper trains it for 100 epochs).
import torch
import torch.nn as nn
import torch.nn.functional as F
d, k = 512, 10 # embedding size, classes
f = nn.Linear(3 * 32 * 32, d) # stand-in frozen encoder
f.requires_grad_(False)
probe = nn.Linear(d, k) # holds W and b
opt = torch.optim.SGD(probe.parameters(), lr=0.1)
x = torch.randn(64, 3 * 32 * 32)
y = torch.randint(0, k, (64,))
for epoch in range(100):
with torch.no_grad():
h = f(x) # [64, d]
loss = F.cross_entropy(probe(h), y)
opt.zero_grad()
loss.backward()
opt.step()
1Frozen encoder f and the probe layer holding W and b, Section 4 eq lines 1 and 2.
2The toy labeled batch, the images x of Section 4 eq line 1 and their labels.
3Features h with no gradient, Section 4 eq line 1.
4Cross entropy on Wh + b, Section 4 eq lines 2 and 3; F.cross_entropy applies the softmax itself, so the code passes raw scores.
5Updates W and b only, one gradient descent step of size lr, Section 4 eq line 3.
Fine-tuning trains every weight on the same cross entropy, so the gradient also reaches f and every embedding h moves. It can start from the probe’s trained layer and uses a smaller step size, since a big step knocks the pretrained features off. It can beat a probe because classes the frozen encoder left mixed cannot be split by a straight line, while fine-tuning moves the embeddings until one can. A probe is still the right choice for measuring what the frozen features already hold, and it costs little; masked image modeling is the case where the gap between the two is large.
= cat= dog= wrong side1line from W, baxes = the 2 numbers in h
step 300, cross entropy 0.32, 5 wrong
Figure 6.1 Probes only move the line; 2 fine-tuning moves h.
5 Self-supervised features come close to supervised ones
Each method pretrains a ResNet-50 (a 50-layer image network) on ImageNet (1.28 million photos, 1000 classes) without labels, for 1000 epochs (SimCLR, BYOL) or 800 (MoCo v2); supervised trains 90 epochs with labels. Then a linear probe is trained on all labels, or BYOL is fine-tuned on 1% of them while supervised trains on that 1% only. Top-1 accuracy is the share of test images whose top class is right.
Figure 7. Top, all labels on a frozen encoder; bottom, 1% of labels.
With every label BYOL lands two percentage points below supervised; with 1% it more than doubles it, which is the real use case.