On this page
Every network here reads an image and outputs class scores, and each is remembered for one block it repeats; each block makes the same running layer, a 3×3 convolution from 32 to 64 channels, cheaper, deeper or more stable.
This page builds on CNN Basics: Convolution, Stride and Pooling, which teaches kernels, stride, padding, pooling and layer cost, and on Receptive Fields, Dilation and Causal Convolutions, which teaches the receptive field and dilation.
AlexNet (Krizhevsky et al. 2012) showed that a deep CNN can beat earlier image methods by a wide margin, and its ReLU activations (which keep positive values and set negative ones to zero) made such a network fast to train. It has 5 convolution layers and 3 fully connected layers, about 60 million weights, trained on ImageNet, a labeled photo set with 1,000 classes, and a variant of it won the 2012 ImageNet challenge (Figure 2). Top-5 error counts an image wrong when the true class is not among the model’s five best guesses; top-1 error counts it wrong unless the best guess is right.
It made ReLU the standard activation for deep CNNs (the name follows Nair and Hinton 2010). In the paper, a four-layer CNN with ReLU reached 25% training error on CIFAR-10 six times faster than the same network with tanh (Figure 3). It also used overlapping pooling, windows 3 wide that step by 2, so neighbors share a cell (Figure 4).
In PyTorch the two pools differ only in the window size.
import torch.nn as nn
# (window, stride)
overlap = nn.MaxPool2d(3, 2)
plain = nn.MaxPool2d(2, 2)
VGG (Simonyan and Zisserman 2015) builds a whole image classifier from one small block and asks how deep it should go. The block is the 3×3 convolution with stride 1 and padding 1; five 2×2 max pools and three fully connected layers finish the network, on 224×224 ImageNet images. Three 3×3 layers see a 7×7 patch, like one 7×7 layer, with fewer weights and two extra ReLUs, the same trade as two 3×3 layers against one 5×5 (Stacking small kernels, in Receptive Fields).
To test how much depth helps, the paper trains configurations A (11 weight layers) to E (19) from the same parts and scores each by top-5 error on the ImageNet validation set, with images resized so the shorter side is 256 pixels for training and testing. A-LRN adds local response normalization, which rescales each value by its neighbors in nearby channels. C is B plus three 1×1 layers, and D puts 3×3 layers in those places.
| Config | Weight layers | Top-5 error |
|---|---|---|
| A | 11 | 10.4% |
| A-LRN | 11 | 10.5% |
| B | 13 | 9.9% |
| C | 16 | 9.4% |
| D | 16 | 8.8% |
| E | 19 | 9.0% |
Top-5 error at a single test scale (the VGG paper, Table 3).
Error falls from A to D and rises at E. In the paper's words, "the error rate of our architecture saturates when the depth reaches 19 layers". At equal depth the all-3×3 D beats C, and LRN does not help.
Figure 5. Top-5 error of configurations A to E; D is lowest.
The same cat can fill a quarter of the frame or all of it, so Inception (Szegedy et al. 2015) does not pick one kernel size. It runs 1×1, 3×3 and 5×5 branches and a pooling branch in parallel and joins their channels.
A 1×1 kernel looks at one pixel, so it cannot find shapes; it only blends the channels at that pixel. The idea comes from Network in Network (Lin et al. 2014).
The same blend runs at every pixel, so the map keeps its grid; flattening the map into a dense layer would lose the positions and need separate weights for every pixel. In the running example a 1×1 from 32 to 64 channels has 32 · 64 = 2,048 weights, against 18,432 (32 · 64 · 9) for the 3×3.
In PyTorch a 1×1 is an ordinary nn.Conv2d with kernel_size=1; the bias b is stored apart from W.
import torch
import torch.nn as nn
mix = nn.Conv2d(32, 64, 1)
print(mix.weight.shape)
# [64, 32, 1, 1]: W[o, c]
print(mix.weight.numel())
# 2048
x = torch.randn(1, 32, 56, 56)
print(mix(x).shape)
# [1, 64, 56, 56]
That cheapness is why Inception puts a 1×1 in front of its 3×3 and 5×5 branches: it shrinks the channel count first, so the large kernels stay affordable. torchvision builds each block as Inception(in, ch1x1, ch3x3red, ch3x3, ch5x5red, ch5x5, pool_proj), where each argument ending in red (reduce) is the 1×1 in front of its kernel. That version adds batch norm after every convolution, which the 2015 network did not have, and its “5×5” branch actually uses a 3×3 kernel, a known bug its source notes and keeps for the trained weights (torchvision googlenet.py).
If one channel's values sit near 0.01 and another's near 100, no single step size suits both. Batch normalization, or batch norm (Ioffe and Szegedy 2015), rescales each channel to mean 0 and spread 1, then lets the network learn its own scale and shift:
Batch norm acts differently in training and at test time. On every training step it normalizes with that batch’s own mean and variance per channel, and also updates a running mean and variance per channel as a moving average that gives each new batch a share momentum of the weight. At test time (model.eval()) it uses the running values, so one image is normalized the same way every time, and the normalization becomes a fixed linear map that can be folded into the layer before it, so it costs nothing at inference.
Per channel it stores two learned numbers, γ and β, and two running values, and in training it adds one pass over the map for the statistics. Small batches make those statistics noisy. Group Norm (Wu and He 2018) normalizes within one image instead, so its error barely changes with the batch size (Figure 7).
A block may also end with dropout (Srivastava et al. 2014), which zeroes each value with probability p during training only (Regularization).
In a typical PyTorch block the convolution has no bias, because batch norm’s shift β does that job.
import torch
import torch.nn as nn
cfg = dict(padding=1, bias=False)
conv = nn.Conv2d(32, 64, 3, **cfg)
block = nn.Sequential(
conv,
nn.BatchNorm2d(64),
nn.ReLU(),
nn.Dropout2d(p=0.1),
)
x = torch.randn(8, 32, 56, 56)
y = block(x)
# train: batch stats, running
# stats updated in place
block.eval()
y = block(x)
# eval: running stats, no drop
bn = block[1]
print(bn.running_mean.shape)
# [64], one per channel
Dropout2d zeroes whole channels with probability p = 0.1.The batch norm paper trains Inception, the network from Section 3, and batch-normalized versions of it on ImageNet. BN-Baseline uses Inception's learning rate of 0.0015 (the step size, see optimizers); BN-x5 and BN-x30 use 5 and 30 times it; BN-x5-Sigmoid swaps ReLU for sigmoid. Batch norm reaches Inception's best accuracy, 72.2%, in 14 times fewer steps.
ResNet (He et al. 2016) adds each block's input back to the block's output, so networks with over a hundred layers still train. With this shortcut, a block that should change nothing only has to push its layers' output to zero, instead of learning an exact copy of its input:
ResNet-50 uses this bottleneck block: a 1×1 shrinks the channels, a 3×3 works on the narrow map, and a 1×1 widens it back. Its receptive field, 427 pixels in torchvision's layout, is worked out in Receptive Fields, Dilation and Causal Convolutions (receptive fields of real networks), and Transformer blocks use the same shortcut (the Transformer block).
A complete ResNet-50 bottleneck as torchvision builds it. Conv is Conv2d with bias=False, since batch norm follows every convolution; self.f is F(x) with i channels in and middle width m; the 3×3 carries the stride s, and the output is 4m wide.
import torch
import torch.nn as nn
from torch.nn import Conv2d
from functools import partial
Conv = partial(Conv2d, bias=False)
def conv_bn(i, o, k, s=1):
c = Conv(i, o, k, s, k // 2)
bn = nn.BatchNorm2d(o)
return nn.Sequential(c, bn)
class Bottleneck(nn.Module):
def __init__(self, i, m, s=1):
super().__init__()
o = 4 * m
self.f = nn.Sequential(
conv_bn(i, m, 1),
nn.ReLU(),
conv_bn(m, m, 3, s),
nn.ReLU(),
conv_bn(m, o, 1))
# default: 1×1 projection
skip = conv_bn(i, o, 1, s)
if s == 1 and i == o:
skip = nn.Identity()
self.skip = skip
def forward(self, x):
h = self.f(x) # F(x)
y = h + self.skip(x)
return torch.relu(y)
block = Bottleneck(64, 64)
x = torch.randn(1, 64, 56, 56)
print(block(x).shape)
# [1, 256, 56, 56]
The shortcut runs on every forward pass and adds no weights when shapes match, only one addition per value; inside the block the 3×3 holds half the weights (Figure 11). ResNet-50 stacks such blocks in four stages (Figure 12). torchvision’s zero_init_residual option starts each block’s last batch norm γ at 0, so every block begins as the identity; its source credits this with 0.2 to 0.3 points of accuracy. torchvision puts the stride on the 3×3, the layout often called ResNet v1.5; the original paper put it on the first 1×1 (torchvision resnet.py).
Most of these failures come from a block on this page used in the wrong setting.
| Symptom | Likely cause | Fix |
|---|---|---|
| Accuracy drops with batches of 2 to 4 images | Statistics from so few images are noisy | Group Norm, which normalizes each image on its own |
| Test results change with the batch | Batch norm still in training mode | Call model.eval() |
| A deeper plain network trains worse | No shortcut around the blocks | Residual blocks |
ResNet-50 holds about 25.6 million weights (Figure 12); how do you keep its accuracy on a phone with a fraction of them? Efficient CNNs: MobileNet and EfficientNet answers that.