On this page
A convolutional network (CNN) reads an image and outputs a few numbers, such as one score per digit. Each layer slides small grids of weights, its kernels, over the input and outputs feature maps of where their patterns appear; pooling and stride shrink those maps, and a final layer turns them into scores.
A convolution layer takes an image and outputs one map per kernel, showing where that kernel's pattern appears. It slides one small grid of weights, the kernel (or filter), across the input and reuses it at every position. At each position the kernel multiplies each weight by the input value under it, adds the products and adds a bias:
A fully connected layer (see the MLP tutorial) instead gives every input value its own weight. For one output on a 224×224 color image with three channels (red, green, blue):
That is about 5,600 times fewer weights, and the reuse is called shared weights.
Reusing one kernel everywhere gives three properties. Shift the input and the output map shifts by the same amount, which is translation equivariance (invariance would mean the output does not change at all). The same 27 weights replace 150,528, which is parameter efficiency. Each output depends on a small patch, because nearby pixels usually belong together, which is locality.
A layer runs many kernels, and each one makes one output channel, also called a feature map. A nonlinearity follows, usually ReLU, which keeps positive values and sets negative ones to zero, so a stack of layers can fit patterns that one weighted sum cannot. A one-dimensional kernel is the same operation as an FIR (finite impulse response) filter in signal processing, a fixed weighted sum of the last few samples.
In PyTorch, nn.Conv2d(3, 16, 3) is one layer of 16 kernels, each 3×3 over the 3 color channels. Each output map is 222 wide: the kernel must fit inside the image, so 1 pixel is lost from each edge.
import torch
import torch.nn as nn
conv = nn.Conv2d(3, 16, 3)
x = torch.randn(1, 3, 224, 224)
print(conv.weight.shape)
# [16, 3, 3, 3]
a = torch.relu(conv(x))
print(a.shape)
# [1, 16, 222, 222]
A kernel scores high where the patch looks like the kernel and low elsewhere. The NumPy slide below, for one channel and a square input, stores the match score z[i, j] of Section 1, without the bias, at every output cell. A k-wide kernel fits n − k + 1 times across an n-wide input.
import numpy as np
def conv2d(x, K):
# x: [n, n], K: [k, k]
n, k = len(x), len(K)
out = n - k + 1
z = np.zeros((out, out))
cells = np.ndindex(out, out)
for i, j in cells:
patch = x[i:i+k, j:j+k]
z[i, j] = np.sum(patch*K)
return z # [out, out]
Figure 3. The vertical edge kernel scores the bar’s two sides with opposite signs.
Three hyperparameters set how large the output map is, by deciding how many positions the kernel visits. The kernel size k is the patch it sees. The stride s is how far it moves each step. The padding p is how many zeros go around the input. Valid padding adds none, and same padding adds enough to keep the output as large as the input.
The NumPy function of Section 1.1 with stride and padding added runs the output size card exactly.
import numpy as np
def conv2d(x, K, s=1, p=0):
x = np.pad(x, p)
# x: [n, n] with its border
n, k = len(x), len(K)
out = (n - k) // s + 1
z = np.zeros((out, out))
cells = np.ndindex(out, out)
for i, j in cells:
r, c = i * s, j * s
patch = x[r:r+k, c:c+k]
z[i, j] = np.sum(patch*K)
return z
Memory follows the maps, not the weights, because training keeps every layer’s output for the backward pass (Figure 4). The full argument list and defaults are in the PyTorch Conv2d docs.
Pooling keeps one number per window, so the map gets smaller and each later output covers more of the image. It has no weights. Max pooling keeps only the largest value, so the other values and that value's exact place in the window are lost.
In PyTorch, each pooling layer below takes 16 maps of 32×32. The 2 is the window, and the stride defaults to the window. A larger window shrinks the map faster and throws away more position; a window of 1 does nothing.
import torch
import torch.nn as nn
x = torch.randn(1, 16, 32, 32)
max_pool = nn.MaxPool2d(2)
avg_pool = nn.AvgPool2d(2)
gpool = nn.AdaptiveAvgPool2d(1)
print(max_pool(x).shape)
# [1, 16, 16, 16]
print(avg_pool(x).shape)
# [1, 16, 16, 16]
print(gpool(x).shape)
# [1, 16, 1, 1]
Average pooling suits audio and time series, where the energy in a window matters more than one peak. Global average pooling, the last line, leaves one number per channel and can replace a large fully connected head, which removes that head's many weights, a common source of overfitting (memorizing the training images); adaptive pooling picks the window itself so any input size gives the output size you ask for.
A convolution with a stride above 1 also downsamples, with learned weights. Both can cause aliasing: detail finer than the new sampling grid folds back as a false coarse pattern (Aliasing and Anti-Aliasing Filters).
A blur before the stride removes that fine detail first. BlurPool (Zhang 2019) does this, and its outputs change less when the input shifts by one pixel; it costs one fixed blur per downsampling layer and no weights. Its official code uses a 4-tap blur by default; a wider blur removes more aliasing but also more real detail. Zero padding adds an artificial dark border; reflect padding mirrors the edge outward instead.
A tiny CNN takes a 28×28 image of a handwritten digit and outputs ten scores, one per digit. Each stage shrinks the maps and adds more of them, so the last stage holds a few numbers that each summarize a large part of the digit. In PyTorch the whole net is one nn.Sequential; each 5×5 convolution trims 4 pixels from each side length and each pool halves it:
import torch.nn as nn
net = nn.Sequential(
nn.Conv2d(1, 6, 5),
nn.ReLU(),
nn.MaxPool2d(2), # 6×12×12
nn.Conv2d(6, 16, 5),
nn.ReLU(),
nn.MaxPool2d(2), # 16×4×4
nn.Flatten(), # 256
nn.Linear(256, 10),
)
Figure 6. Real shapes; the kernels and weights are fixed, not learned.
A 28×28 handwritten 7. Everything the network knows starts here.
The walkthrough net is a cut-down sketch with 5,142 weights. Its full-size ancestor, LeNet-5 (LeCun et al. 1998), takes 32×32 inputs, adds a third convolution layer and an 84-unit layer, and has about 60,000 trainable weights. Trained on MNIST's 60,000 handwritten digits and tested on 10,000 others, it misclassifies under 1% of the test digits (Figure 7). Every digit at every position trains the same kernels, which is why so few weights suffice.
Both problems come from how a layer samples and stores its maps.
| Symptom | Likely cause | Fix |
|---|---|---|
| A one-pixel shift changes the output | Stride without a blur aliases fine detail | Blur before downsampling (BlurPool) |
| Out of memory on large inputs | Activation maps, not weights, fill memory in training | Downsample earlier or use a smaller batch |
How much of the image does one output see, and how do you grow that view without more weights? Receptive Fields, Dilation and Causal Convolutions answers that.