← All tutorials

Self-Supervised Learning Failure Modes and Checks

On this page

Every method here learns embeddings (output vectors) and fails when it can solve its training task without learning the objects, their parts and the scene.

The stop-gradient and predictor are taught in Non-Contrastive Self-Supervised Learning, and the 75% mask in Masked Image Modeling. Image-Text Contrastive Learning (CLIP) uses the other captions in its batch as negatives, so a small batch gives it the “Too few negatives” failure too.

1 Overlapping receptive fields give the network a shortcut

A receptive field is the part of the input an output sees. When the inputs seen by the two compared outputs overlap or touch, the network can match the shared pixels and learn nothing about the objects in the image.

1.1 Patch position needed a gap, jitter and no color

Doersch et al. (2015) asked a network which of eight positions a second 96 px patch sat at around the first. Shared edges, continuing lines and a lens’s color fringe (chromatic aberration) gave it away, so they added a gap, a random shift (jitter) and dropped color.

1.2 Predicting further ahead leaves the overlap behind

CPC (Contrastive Predictive Coding, van den Oord et al. 2018) learns from speech: from the audio so far it predicts summaries of the next K short stretches. The nearest stretches overlap the current one, so predicting them can copy shared samples. The accuracy of a linear classifier at naming the phoneme (speech sound, 41 classes, LibriSpeech) from the learned vectors shows which K learns more.

2 Collapse puts embeddings on one point or a few directions

Complete collapse maps every input to one embedding; the loss reaches its lowest value while accuracy falls to chance, as SimSiam does when its stop-gradient (a block on gradients to the target side) is removed. In dimensional collapse (Jing et al. 2022), the embeddings spread along only a few directions, so a linear probe scores poorly.

Negatives do not prevent it: the dimensional-collapse paper (Fig. 2) finds that SimCLR’s 128-number embeddings leave several directions unused. It has two causes. Edits that change images along a direction more than real images differ along it teach the encoder to ignore that direction (sec. 4). Stacked layers also pull their weights toward few directions (implicit regularization), even under weak edits (sec. 5).

3 Batch size, temperature and augmentation decide success

Three settings decide whether contrastive training works. Batch size sets how many negatives (images pushed away) SimCLR sees. The temperature τ sets how sharply the loss focuses on the closest negatives. Augmentation sets what the two views share.

4 BatchNorm can leak information across a batch

BatchNorm normalizes each feature with the mean and variance of the samples on one GPU. In MoCo, the query is one view’s embedding and its positive key is the other view’s. If the two sit in the same GPU batch, they share those statistics, and the network can match them by that shared signal instead of by content (MoCo App. A.9). MoCo shuffles keys across GPUs; SimCLR averages statistics over all devices (SyncBatchNorm on GPUs).

5 A probe score is not the whole story

A probe and fine-tuning (training the encoder too) can rank methods differently. MoCo v2 tested two changes on MoCo: an MLP projection head and blur augmentation (Table 1, ResNet-50, 200 epochs). The head helps the frozen-encoder probe more; blur helps more once the encoder is fine-tuned for object detection (drawing boxes around objects, PASCAL VOC).

6 Five checks to watch while training

Five cheap checks catch these failures early. Alignment (how close pairs sit) and uniformity (how evenly embeddings spread) come from Wang and Isola’s paper.

The first snippet checks one batch of d-number embeddings; eff_rank counts the directions in use (Roy and Vetterli 2007).

import torch
import torch.nn as nn
import torch.nn.functional as F

N, d = 256, 128
encoder = nn.Linear(32, d)                       # stand-in
x = torch.randn(N, 32)
with torch.no_grad():
    Z = F.normalize(encoder(x), dim=1)           # [N, d], unit rows
spread = Z.std(dim=0).mean()                     # near 1/sqrt(d)
C = torch.cov(Z.T, correction=0)                 # divide by N
lam = torch.linalg.svdvals(C)
shares = lam / lam.sum()
eff_rank = torch.special.entr(shares).sum().exp()   # 1 to d

1. 1One batch of unit embeddings Z, the z_i of Section 2 eq line 1.
2. 2Spread per dimension, the first row of Figure 11.
3. 3Covariance C, eq lines 1 and 2.
4. 4Singular values and the effective rank, the bars in Figure 6.

The second follows that paper’s code; both numbers fall as training works.

import torch
import torch.nn.functional as F

ALPHA, UNIF_T = 2, 2                             # the paper's defaults
N, d = 256, 128
z1 = F.normalize(torch.randn(N, d), dim=1)
z2 = F.normalize(z1 + 0.1 * torch.randn(N, d), dim=1)
align = (z1 - z2).norm(dim=1).pow(ALPHA).mean()
sq = torch.pdist(z1).pow(2)                      # every pair, squared
uniform = torch.exp(-UNIF_T * sq).mean().log()   # at most 0

1. 1Toy positive pairs, two views per image, the pairs measured in the alignment row of Figure 11.
2. 2Alignment, the positive-pair distance, the third row of Figure 11. A larger ALPHA weights pairs that sit far apart more; at 2 it is the squared distance.
3. 3Uniformity over all pairs, the fourth row of Figure 11. A larger UNIF_T scores only the closest pairs; a smaller one scores all pairs evenly. The paper’s best runs used t = 2 (STL-10, NYU-Depth-V2), 3 (ImageNet-100) and 5 (BookCorpus).

The third is k-NN accuracy, a majority vote of the 20 nearest training embeddings.

import torch
import torch.nn.functional as F

K_NEIGHBORS, n_train, n_val, d = 20, 1000, 200, 128
bank = F.normalize(torch.randn(n_train, d), dim=1)
train_y = torch.randint(0, 10, (n_train,))
query = F.normalize(torch.randn(n_val, d), dim=1)
val_y = torch.randint(0, 10, (n_val,))

sim = query @ bank.T                             # [n_val, n_train]
top = sim.topk(K_NEIGHBORS, dim=1)
pred = train_y[top.indices].mode(dim=1).values   # majority vote
knn_acc = (pred == val_y).float().mean()

1. 1A bank of labeled training embeddings and queries to classify, for the last row of Figure 11.
2. 2Cosine, top 20, majority vote and accuracy, the last row of Figure 11. More neighbors give a smoother but blurrier vote; at 1 a single noisy neighbor decides. DINO found 20 neighbors best with a similarity-weighted vote; a plain majority vote is the simplest version.

Knowledge Distillation, a trained encoder teaching a smaller student, is a related tutorial.

7 References

• Doersch, C., et al. (2015). Unsupervised Visual Representation Learning by Context Prediction. ICCV 2015. arXiv:1505.05192.
• van den Oord, A., et al. (2018). Representation Learning with Contrastive Predictive Coding. Preprint. arXiv:1807.03748.
• Chen, T., et al. (2020). A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020. arXiv:2002.05709.
• He, K., et al. (2020). Momentum Contrast for Unsupervised Visual Representation Learning. CVPR 2020. arXiv:1911.05722.
• Chen, X., et al. (2020). Improved Baselines with Momentum Contrastive Learning. Preprint. arXiv:2003.04297.
• Grill, J.-B., et al. (2020). Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. NeurIPS 2020. arXiv:2006.07733.
• Chen, X., and He, K. (2021). Exploring Simple Siamese Representation Learning. CVPR 2021. arXiv:2011.10566.
• Caron, M., et al. (2021). Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021. arXiv:2104.14294.
• He, K., et al. (2022). Masked Autoencoders Are Scalable Vision Learners. CVPR 2022. arXiv:2111.06377.
• Jing, L., et al. (2022). Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR 2022. arXiv:2110.09348. (Term from Hua et al. 2021.)
• Roy, O., and Vetterli, M. (2007). The Effective Rank: A Measure of Effective Dimensionality. EUSIPCO 2007.
• Wang, T., and Isola, P. (2020). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML 2020. arXiv:2005.10242.