On this page
Every output of a CNN sees only a limited window of its input, its receptive field. Stacking layers and striding widen that window, spreading a kernel’s taps apart (dilation) widens it with no extra weights, and for a live signal the window must reach only into the past.
This page builds on CNN Basics: Convolution, Stride and Pooling, which slides a kernel over the input with a stride and keeps one value per patch with pooling.
A neuron's receptive field (RF) is the patch of input pixels that can change its value; this page also calls it the window. Early layers see edges; deep layers can see whole objects, but only if the window is large enough. Section 1.3 and Section 1.5 use two well-known image networks. VGG-16 is a deep stack of 3×3 convolutions, 13 of them in blocks of 2, 2, 3, 3 and 3 with pooling between blocks (VGG, in Classic CNN Architectures). ResNet-50 is 16 bottleneck blocks with shortcuts, and each block’s two 1×1 layers mix channels without widening the window (ResNet, in Classic CNN Architectures).
One rule, applied layer by layer from the input up, gives the window of any network. It tracks two numbers: RF, and the jump J, how far apart two neighboring outputs sit, in input pixels. Both start at 1. For a layer with kernel k and stride s (the dilation d, the spacing between a kernel's taps, is 1 until Section 2):
Two stacked 3×3 layers have a receptive field of 5, the same as one 5×5 layer, with 18 weights per channel pair instead of 25 and an extra ReLU between them.
A pool with stride 2 barely widens the window itself, but it doubles J, so every later layer grows twice as fast. In the first block of a typical CIFAR model (CIFAR is a set of 32×32 color photos), the 3×3 after the pool adds twice what the same 3×3 added before it (Figure 4).
Pools and strides keep multiplying J, so real networks reach hundreds of pixels.
| Network | Layers that grow the window | RF |
|---|---|---|
| LeNet-style | conv 5×5, pool 2, conv 5×5, pool 2 | 16 |
| VGG-16 | 13 convs 3×3 in blocks of 2, 2, 3, 3, 3, a stride-2 pool after each block | 212 |
| ResNet-50 | stem 7×7 stride 2, pool 3×3 stride 2, 16 blocks each with one 3×3 (stride 2 at three stage starts) | 427 |
ResNet-50's number depends on where each stage puts its stride. With the stride on the 3×3, as in torchvision, the recurrence gives 427; other layouts give 483, the value Araujo et al. (2019) report.
Figure 5. The ▲ marks the window’s reach after each layer.
Pixels near the window's center reach the neuron through many paths and edge pixels through few, so influence falls off like a bell (Luo et al. 2016). A rough model treats influence as a bell over the distance from the center.
Whether a network sees an object whole comes down to one comparison: the object's width against the effective receptive field, not only against the whole receptive field. The function below makes that check for an object obj pixels wide, against the windows of LeNet (16), VGG-16 (212) and ResNet-50 (427):
# share of RF above half the peak
ERF = 0.471
def verdict(rf, obj):
# rf, obj: widths in pixels
if obj <= ERF * rf:
return "fully seen"
if obj <= rf:
return "partly seen"
return "not covered"
for rf in (16, 212, 427):
print(verdict(rf, obj=150))
Figure 6. The verdict flips as orange outgrows the band, then the window.
Overlapping windows matter in self-supervised learning too (SSL failure modes).
Spread the kernel's taps apart, raising the dilation d, and one kernel covers more input with the same weights (Yu and Koltun 2016):
Stacking the same dilation repeats the same holes, so the top output never sees some inputs (the gridding problem, Wang et al. 2018). Mixing rates such as 1, 2, 4 fills them for a 3-tap kernel. A stride before dilated layers spreads the taps further and makes the holes worse, so a plain convolution every few layers restores full coverage.
A dilated first layer skips raw samples before anything has summarized them, so dilation usually starts after a few plain layers. Running several dilated convolutions side by side at different rates and joining their outputs sees fine and coarse context at once; DeepLab calls this Atrous Spatial Pyramid Pooling, or ASPP (Chen et al. 2018, rates 6, 12, 18 and 24; atrous means dilated). The ASPP-style block below runs B branches on a map x with C channels. The positional arguments of Conv2d are in, out, kernel, stride, padding and dilation. Each branch costs as much as a plain 3×3, so the block runs about B times the compute of one layer.
import torch
from torch.nn import Conv2d
from torch.nn import ModuleList
C, rates = 64, (1, 2, 4)
branches = ModuleList()
for d in rates:
# padding d keeps the size
b = Conv2d(C, C, 3, 1, d, d)
branches.append(b)
B = len(rates)
fuse = Conv2d(B * C, C, 1)
x = torch.randn(1, C, 32, 32)
outs = [b(x) for b in branches]
y = fuse(torch.cat(outs, dim=1))
# y: [1, C, 32, 32]
At 1,000 samples per second a 300 ms event spans 300 samples, so the last layer’s window must reach at least 300. The function below runs the recurrence of Section 1 over (k, s, d) tuples, with rf and jump as RF and J, and scores the seven-layer design of Figure 9 and a copy whose last dilation doubles once more:
def receptive_field(layers):
rf, jump = 1, 1 # RF and J
for k, s, d in layers:
rf += (k - 1) * d * jump
jump *= s
return rf
plain = [(7, 1, 1), (5, 2, 1)]
dils = (2, 4, 8, 16, 16)
stack = [(5, 1, d) for d in dils]
design = plain + stack
print(receptive_field(design))
# 379
wider = design[:-1] + [(5, 1, 32)]
print(receptive_field(wider))
# 507
When the 300 ms detector of Section 2.1 runs live, at each new sample it must decide using only past samples, but a kernel with centered padding reads half its reach from the future. A Conv1D slides a kernel along one axis, time. In a causal task the output at time t cannot use later inputs, so the padding goes on the left only:
A causal convolution in PyTorch is a plain nn.Conv1d that pads (k − 1) · d zeros on the left, so the output keeps the input's length and never sees a later step.
import torch
import torch.nn as nn
import torch.nn.functional as F
class CausalConv1d(nn.Conv1d):
"""Looks back only"""
def forward(self, x):
# x: [B, C, T]
k = self.kernel_size[0]
d = self.dilation[0]
pad = (k - 1) * d
# pad the past side only
x = F.pad(x, (pad, 0))
return super().forward(x)
A model trained with ordinary centered padding peeks at the future and scores well offline, then fails live. Doubling the dilation layer by layer grows the window exponentially, as in Section 2. A temporal convolutional network (TCN) stacks such layers:
If the dilation grows too fast, some time steps fall into holes the output never sees, the gridding problem (Figure 8); repeating a cycle such as 1, 2, 4, 1, 2, 4 avoids it. Deep stacks also need a shortcut so gradients reach the early layers: each level adds its input back to its output, the residual shortcut ResNet introduced (ResNet, in Classic CNN Architectures). One TCN level in PyTorch takes a ready-built CausalConv1d with dilation d. Dropout zeroes each value with probability p while training and keeps every value at test time (Regularization). Raising p regularizes harder, p = 0 turns it off and near 1 almost nothing survives; the TCN code uses 0.2. The authors’ code is at github.com/locuslab/TCN; stacking levels with d = 1, 2, 4 and so on gives the card's window:
class TCNLevel(nn.Module):
"""y = relu(x + branch)"""
def __init__(self, conv):
super().__init__()
self.conv = conv
drop = nn.Dropout(0.2)
self.drop = drop
def forward(self, x):
# x and y: [B, c, T]
h = self.conv(x)
h = torch.relu(h)
h = self.drop(h) # branch
return torch.relu(x + h)
c, k, d = 16, 3, 4
kw = dict(dilation=d)
conv = CausalConv1d(c, c, k, **kw)
level = TCNLevel(conv)
For multi-sensor data such as EEG (recordings of brain waves), treat the sensors as the height of a sensors-by-time grid and apply a Conv2D whose kernel spans all sensors and a time window; for live use the causal padding then goes on the left of the time axis only. The Conformer mixes convolutions with attention for speech (The Conformer Block).
A convolution sees a fixed window. Recurrent networks (RNN Basics) carry a running state through time, and attention (Attention) lets every position see every other one in a single layer.
Most of these failures come from a receptive field that is smaller than the object, an effective receptive field smaller than the object, dilation gaps that skip inputs, or padding that lets a live model see the future.
| Symptom | Likely cause | Fix |
|---|---|---|
| Large objects are missed | Receptive field smaller than the object | Pool or stride earlier, or add dilation |
| Large objects are detected weakly or inconsistently | Object extends past the effective receptive field | Keep objects under half the receptive field |
| Fine detail vanishes after dilated layers | Dilated taps skip inputs (gridding) | Mix rates such as 1, 2, 4 |
| Offline scores beat live ones | Centered padding saw the future | Causal, left-only padding |
Which networks stacked these layers and won ImageNet? Classic CNN Architectures: AlexNet to ResNet answers that.