How a network works out every weight’s share of the error, and how to check it.
On this page
Gradient descent moves every weight against the slope of the loss, so it needs one slope per weight. Backpropagation gets all of them in one backward pass. It applies the chain rule from the loss back to the input and reuses the values the forward pass stored. Gradient checking compares that answer with a slow nudge-and-measure estimate.
Everything here runs on the small network of Multilayer Perceptrons, on its blue point x = (−2, 2) with label y = 1. The input x goes through the first layer’s weights W1 and biases b1 to the hidden sums z. ReLU turns them into the hidden values h. The second layer’s weights W2 and bias b2 give the output score s, and the sigmoid turns s into the probability ŷ.
Every training loop repeats the same four moves.
The backward pass runs once per training step, after the forward pass and the loss and before the optimizer step. It costs about twice the forward pass. For a language model, Scaling Laws counts about 6 FLOPs (floating-point operations) per parameter for each training token. Two are in the forward pass and four in the backward pass. Gradient checking runs once, when you write a new layer or loss, never inside the loop.
A weight’s slope can be measured directly: nudge the weight and watch the loss. The second layer has weights W2 = (0.5, 0.5); nudge its first entry.
| W2 entry 1 | loss L | slope, change over 0.001 |
|---|---|---|
| 0.5 | 0.38687 | −0.641 |
| 0.501 | 0.38623 | |
| backprop, Section 4.1 | −0.642 | |
This network has 9 weights and biases: 4 in W1, 2 in b1, 2 in W2 and 1 in b2. Nudging needs 9 extra forward passes, or 18 for the two-sided check of Section 7. Backpropagation gets all 9 slopes in one backward pass that costs about two forward passes. A model with billions of weights could never be trained by nudging.
The network’s last two steps turn the score s into the probability ŷ with the sigmoid σ, then ŷ into the loss, −log ŷ for label 1.
| step | at s = 0.75 |
|---|---|
| the sigmoid gives ŷ | 0.679 |
| slope of the loss along ŷ, which is −1 over ŷ | −1.472 |
| slope of the sigmoid along s, which is ŷ times 1 minus ŷ | 0.218 |
| their product | −0.321, exactly the probability minus the label |
A value that feeds the loss along several paths gets the sum of their gradients.
| one value, several paths | its gradient |
|---|---|
| shared layers under two task outputs (Multitask Learning) | (2, 0) and (−1, 0.5) add to (1, 0.5), illustrative |
| a residual step, which adds a block F’s output to the block’s own input, with F′ the slope of F (The Transformer Block) | 1 plus F′, so 1.2 when F′ is 0.2, illustrative |
| an RNN’s weight U, shared by every time step (RNN Basics) | one term per step; 0.3, 0.1 and 0.05 add to 0.45, illustrative |
This is not the chain rule of Probability and Bayes Rule, which multiplies probabilities, not slopes.
For every value the forward pass stored, the backward pass computes an error, named δ (delta). δs is the slope of the loss L along s.
| gradient | from the errors of Figure 2 | on the blue point |
|---|---|---|
| W2 | δs times h | (−0.642, 0) |
| b2 | δs itself | −0.321 |
| W1 | δz times x as a row | row 1 (0.321, −0.321), row 2 zeros |
| b1 | δz itself | (−0.160, 0) |
Figure 3. Line width is a weight’s gradient, ring size a value’s error δ.
PyTorch’s autograd records every operation on a tensor marked requires_grad during the forward pass, then walks that record backward. On the blue point it prints the gradients of Section 4.1.
import torch
x = torch.tensor([-2.0, 2.0]) # the blue point, label 1
W1 = torch.tensor([[-1., 1.], [1., -1.]], requires_grad=True)
b1 = torch.tensor([-2.0, -2.0], requires_grad=True)
W2 = torch.tensor([0.5, 0.5], requires_grad=True)
b2 = torch.tensor(-0.25, requires_grad=True)
s = W2 @ torch.relu(W1 @ x + b1) + b2 # the score
loss = torch.nn.functional.softplus(-s) # BCE for label 1, 0.3869
loss.backward()
print(W1.grad, b1.grad, W2.grad, b2.grad)
# W1.grad [[0.3208, -0.3208], [0, 0]] b1.grad [-0.1604, 0]
# W2.grad [-0.6416, 0] b2.grad -0.3208requires_grad=True so autograd records every operation on them.softplus is a smooth version of ReLU, the log of 1 plus e to its input. softplus(-s) is the same loss as −log ŷ, worked out from the score so it never takes the log of 0 (Section 8). backward walks the record back. This runs once every training step.Two controls turn the record off. This snippet reuses the tensors of the autograd snippet.
target = loss.detach()
with torch.no_grad(): # nothing recorded
y_val = torch.sigmoid(W2 @ torch.relu(W1 @ x + b1) + b2)
print(loss.requires_grad, target.requires_grad, y_val.requires_grad)
# True False Falsedetach gives the same value with no record behind it, so no gradient flows back through it: a fixed target, as in GradNorm.no_grad records nothing inside its block, for evaluation.The values the forward pass stores are the activation memory. Checkpointing (Chen et al., 2016) stores fewer and recomputes the rest during the backward pass; CLIP uses it to fit its huge batches.
The walk of Section 4.1 multiplies one local slope per step, so a long chain multiplies many.
| slope per step a | after n = 27 steps back |
|---|---|
| 0.90, the default of RNN Basics, section 2.1, whose 28 time steps are 27 steps back to step 1 | 0.058, about 6% |
| 1.1, an illustrative slope (Figure 4 can set it) | 13.1 |
Figure 4. The error left after each step back, on a log scale.
Each fix keeps the factors near 1 or gives the gradient a path around them.
| fix | what it does to the product |
|---|---|
| ReLU (MLP, section 2) | slope exactly 1 where a unit fires, while the sigmoid’s slope is at most 0.25 |
| variance-keeping init (Glorot and Bengio, 2010; He et al., 2015) | sets each layer’s starting weights so its outputs keep their variance, and in the 2010 scheme its gradients too, so the factors start near 1 |
| residual connections (ResNet) | the skip adds 1 to each layer’s slope, so one path skips every layer’s factor |
| gating (LSTM) | the cell state is updated by adding; the factor along it is the forget gate, a learned number between 0 and 1 for how much of the old cell state to keep, which can stay near 1 |
| normalization (Normalization Layers) | keeps each layer’s values on one scale, so the factors stay near 1 |
| clipping (Optimizers in Practice) | caps the gradient’s size, for the exploding end only |
Section 2’s one-sided nudge read −0.641 where backprop gives −0.642. Nudging both ways closes most of that gap, and a relative score says when two slopes agree.
The table runs both nudges on W1 entry (1, 1), whose backprop slope is 0.320821.
| W1 entry (1, 1), float64 | one-sided r | central r |
|---|---|---|
| ε = 10−2 | 3.4 × 10−3 | 4.1 × 10−6 |
| ε = 10−4 | 3.4 × 10−5 | 4.0 × 10−10 |
PyTorch’s torch.autograd.gradcheck nudges by ε = 10−6 in float64 and passes an entry when the gap between the two slopes is small. On W1 entry (1, 1), whose slope is 0.32, a gap up to about 0.00033 passes.
Both central rows of the table pass with room. A bug looks very different: a dropped factor of 2 gives r = 0.5, and a wrong sign gives r = 2.
Turn off randomness first, such as dropout, which zeroes random units during training (Regularization, section 1.1). Stay away from ReLU’s kink at 0, where the nudge sees a slope backprop does not; Figure 3 shows the kink. The same check runs in code on the same weight.
import torch
f64 = dict(dtype=torch.float64)
x, b1 = torch.tensor([-2., 2.], **f64), torch.tensor([-2., -2.], **f64)
W2, b2 = torch.tensor([0.5, 0.5], **f64), torch.tensor(-0.25, **f64)
def L(W1): # the loss as a function of W1 alone
return torch.nn.functional.softplus(-(W2 @ torch.relu(W1 @ x + b1) + b2))
W1 = torch.tensor([[-1., 1.], [1., -1.]], **f64, requires_grad=True)
L(W1).backward()
eps, E = 1e-4, torch.tensor([[1., 0.], [0., 0.]], **f64) # E picks entry (1, 1)
g_num = (L(W1.detach() + eps * E) - L(W1.detach() - eps * E)) / (2 * eps)
print(f'{W1.grad[0, 0]:.6f} {g_num:.6f}', torch.autograd.gradcheck(L, (W1,)))
# 0.320821 0.320821 TrueTrue when all pass.| bug | what you see | fix |
|---|---|---|
no zero_grad | gradients of several steps add up, the way two paths add in Section 3 | call zero_grad() at the top of each step |
| an in-place change, such as adding 1 to h by overwriting it, on a value autograd stored | an error at backward: a needed value was modified | add into a new value instead of overwriting h |
.item() or .numpy() inside the loss | an error, or a cut graph: the weights before it get no gradient | keep the loss in tensors |
| a dead ReLU | a unit whose sum stays below 0 never gets a gradient again | MLP training failures |
| log of 0 | the loss is inf, then the gradients and every weight turn NaN (not a number) | compute the loss from the score s, not from ŷ |
The next page, CTC Loss, builds a loss for speech and handwriting, where the labels are not lined up with the input frames, and backpropagation trains through it the same way.
torch.autograd; torch.autograd.gradcheck; torch.utils.checkpoint.