On this page
ResNet-50 holds about 25.6 million weights. The networks here give up a few points of accuracy, or none, for far fewer weights and multiply-adds, small enough for a phone. Four ideas do it: split a convolution into a cheap spatial step and a cheap channel step, invert the residual block so the cheap step works on a wide map, reweight channels with a small gate, and scale depth, width and image size together.
This page builds on Classic CNN Architectures: AlexNet to ResNet, which teaches the 1×1 convolution, batch norm and the residual shortcut that the blocks here reuse.
A standard kernel filters space and mixes channels in one step; doing the two jobs one after the other costs far fewer weights. MobileNet (Howard et al. 2017) builds a whole network from this depthwise separable convolution, an idea used earlier by Sifre (2014) and in Xception (Chollet 2017), and ends with global average pooling and one fully connected layer.
The depthwise step gives every input channel its own 3×3 kernel and never mixes channels. The pointwise step is a 1×1 convolution, which blends the channels at each pixel and sees no neighbors (1×1 convolution, in Classic CNN Architectures), so it does all the mixing. In PyTorch, with ci and co the Cin = 32 and Cout = 64 channels, groups=ci is what makes dw depthwise:
import torch
from torch.nn import Conv2d
from functools import partial
Conv = partial(Conv2d, bias=False)
ci, co = 32, 64
opts = dict(padding=1, groups=ci)
dw = Conv(ci, ci, 3, **opts)
pw = Conv(ci, co, 1)
x = torch.randn(1, ci, 56, 56)
y = pw(dw(x))
print(y.shape)
# [1, 64, 56, 56]
D = dw.weight.numel() # 288
P = pw.weight.numel() # 2048
The split replaces one large weight count with two small ones, so the running example needs about eight times fewer weights.
Figure 4. The gap widens with kernel size and output channels; inputs stay at 32.
MobileNetV2 (Sandler et al. 2018) builds its block from the same split convolution and adds a ResNet-style shortcut around it, so the block adds its input back to its output (ResNet, in Classic CNN Architectures): a 1×1 widens the channels, the depthwise 3×3 filters the wide map cheaply, and a 1×1 narrows it back.
④ The shortcut adds the block’s input to the projected output, so the block only has to learn a change.
The expansion ratio t sets the middle width, t times the input channels. Raising it gives the depthwise filter more channels at proportionally more cost; t = 1 skips the widening 1×1. The paper and torchvision use t = 6 in every stage but the first. The last 1×1 has no ReLU (which keeps positive values and sets negative ones to zero; Convolution, in CNN Basics), because a ReLU on the narrow output would erase information, and the shortcut is used only when the stride (the step between kernel positions; Size, stride and padding, in CNN Basics) is 1 and the width is unchanged.
In PyTorch, branch builds P(D(E(x))), widening with ReLU6 (a ReLU capped at 6), filtering depthwise and projecting back with no activation. The example is a stride-1 block of the 24-channel stage.
import torch
import torch.nn as nn
from torch.nn import Conv2d
from functools import partial
Conv = partial(Conv2d, bias=False)
BN = nn.BatchNorm2d
def branch(c, t):
h = c * t # wide middle
# groups=h: depthwise 3×3
dw = Conv(h, h, 3, 1, 1, 1, h)
return nn.Sequential(
Conv(c, h, 1), BN(h),
nn.ReLU6(),
dw, BN(h), nn.ReLU6(),
Conv(h, c, 1), BN(c))
class InvRes(nn.Module):
def __init__(self, c, t=6):
super().__init__()
self.f = branch(c, t)
def forward(self, x):
return x + self.f(x)
x = torch.randn(1, 24, 56, 56)
print(InvRes(24)(x).shape)
# [1, 24, 56, 56]
The positional arguments of Conv are in, out, kernel, stride, padding, dilation and groups. BN is batch norm, which rescales each channel to mean 0 and spread 1 over the batch, then applies a learned scale and shift (Batch normalization, in Classic CNN Architectures). ImageNet is a labeled photo set with 1,000 classes, and top-1 accuracy is the share of test photos whose highest-scoring class is the right one (AlexNet, in Classic CNN Architectures). The full torchvision model needs a fraction of ResNet-50’s weights and compute for a few points of top-1 (Figure 6; torchvision mobilenetv2.py, torchvision resnet50 weights).
Squeeze-and-excitation (Hu et al. 2018) is a small add-on that scores each channel of a feature map from its average and rescales the channel by that score, so useful channels get louder and others quieter; many CNN blocks carry it, including the B0 blocks of EfficientNet (Section 4).
In the block below, for maps x of shape [B, C, H, W] (batch, channels, height, width), the mean squeezes each map to one number s, two small linear layers turn those into a 0 to 1 weight w per channel, and the last line rescales the maps. The ratio r sets the gate’s hidden width C/r: raising it shrinks the gate and its cost, lowering it gives the gate more capacity, and r = 1 keeps the full width; the paper uses 16.
import torch
import torch.nn as nn
class SEBlock(nn.Module):
def __init__(self, C, r=16):
super().__init__()
self.gate = nn.Sequential(
nn.Linear(C, C // r),
nn.ReLU(),
nn.Linear(C // r, C),
nn.Sigmoid())
def forward(self, x):
s = x.mean((2, 3))
# s: [B, C]
w = self.gate(s) # 0 to 1
w = w[:, :, None, None]
return x * w
x = torch.randn(2, 64, 28, 28)
print(SEBlock(64)(x).shape)
# [2, 64, 28, 28]
It runs on every forward pass, in training and at test time. Added to every ResNet-50 block with r = 16, it adds almost no compute but about a tenth more weights, most of them in the last, widest stage (Figure 8). The authors’ code is at github.com/hujie-frank/SENet.
EfficientNet (Tan and Le 2019) makes a small network larger by raising one coefficient that grows depth, width and image size together. A larger image needs more layers to widen the receptive field (the window of input one output sees; Receptive field, in Receptive Fields) and more channels to hold its finer detail, so growing only one of the three leaves the other two too small for it. It starts from a small network, B0, found by architecture search (neural architecture search) and built from MobileNetV2 blocks (Section 2).
At the same compute, growing all three beats growing any one of them (Figure 9).
B0 is then grown with one coefficient, φ (phi). The three bases come from a small grid search on B0 at φ = 1:
Compound scaling is applied once, at design time: φ picks a model before training and nothing changes while it trains. Raising φ by 1 about doubles the compute, and φ = 0 is B0 itself; the bases 1.2, 1.1 and 1.15 stay fixed. Raising φ step by step gives the family B0 to B7.
| Model | Top-1 | Params | FLOPs |
|---|---|---|---|
| B0 | 77.1% | 5.3M | 0.39B |
| B1 | 79.1% | 7.8M | 0.70B |
| B2 | 80.1% | 9.2M | 1.0B |
| B3 | 81.6% | 12M | 1.8B |
| B4 | 82.9% | 19M | 4.2B |
| B5 | 83.6% | 30M | 9.9B |
| B6 | 84.0% | 43M | 19B |
| B7 | 84.3% | 66M | 37B |
B0 to B7 by ImageNet top-1 accuracy (the paper, Table 2); the released models round these sizes.
torchvision stores a width and a depth multiplier per model (B1 1.0 and 1.1, B4 1.4 and 1.8, B6 1.8 and 2.6), and rounds layer counts up and channels to multiples of 8 (torchvision efficientnet.py).
from torchvision.models import efficientnet_b4
# random weights, nothing is downloaded
model = efficientnet_b4(weights=None)
B7 matches GPipe (Huang et al. 2019) at 84.3% top-1 with about 8 times fewer weights (Figure 10).
At the small end, B1 is 7.6 times smaller and 5.7 times faster than ResNet-152 (the EfficientNet paper). The paper reports B0 and B1 a little differently in different tables (Figure 11).
Other ways to make a model cheaper are quantization, pruning and knowledge distillation.
Each failure here comes from a cheap block used where its saving or its design does not hold.
| Symptom | Likely cause | Fix |
|---|---|---|
| A separable layer saves little | Few output channels, or a 1×1 kernel, where the split adds a layer and saves nothing | Use a standard convolution, as MobileNet does for its first layer and its 1×1s |
| Accuracy drops after the narrow projection | A ReLU on the low-channel output discards information | No activation on the projection |
| Scaling one dimension stops paying off | Depth, width or resolution alone saturates | Scale all three with one φ |
How can a network carry memory across a sequence of any length? RNN Basics: Recurrence and Backpropagation Through Time answers that.