On this page
A recurrent neural network (RNN) reads a sequence one step at a time and carries a hidden state, its memory, from each step to the next. It learns by unrolling that loop over the whole sequence and backpropagating through every step. The same repeated multiplication that lets it remember makes its gradients fade or blow up over long sequences.
| Part | What it answers |
|---|---|
| Order matters | why a sequence needs memory of earlier steps |
| One equation, reused every timestep | how one cell mixes the new input with its memory |
| Backpropagation through time unrolls the loop | why the gradient shrinks or blows up over many steps |
| Why gradients vanish or explode | why a product of many per-step factors fades or blows up |
| Clipping keeps the direction and caps the size | how to cap a gradient that blows up |
| What goes wrong when you actually train one | what breaks in practice |
| Gated RNNs: GRU and LSTM | how gates keep memory alive across many steps |
A feedforward network takes one fixed-size input, multiplies it by weights and applies non-linearities (functions that bend their input, so the network can learn curved patterns). Flatten a sentence, a price series or an audio clip into that one input and which step came when is lost, and sequences of different lengths need padding (dummy steps) or truncating.
With recurrence, you process one input at a time and carry forward some memory, the hidden state h. In code, W and U are learned weight matrices shared by every step, and one more matrix V turns the memory into the prediction. Each step is one dense layer, the building block of an MLP, applied again and again:
import numpy as np
# memory starts empty
h = np.zeros(hidden_dim)
# one input at a time, in order
for x_t in sequence:
# mix input and memory
h = np.tanh(W @ x_t + U @ h)
# memory becomes the prediction
y = V @ h
PyTorch runs the same loop in one call. nn.RNN returns the hidden state at every step and the last one:
import torch
from torch import nn
# x: (batch, T, d_in)
rnn = nn.RNN(
d_in, d_h,
batch_first=True)
out, h = rnn(x)
# out: (batch, T, d_h)
# h: (1, batch, d_h)
d_in to hidden size d_h; batch_first=True puts the batch first and time second in x.Each step mixes today's input with yesterday's memory and squashes the mix with tanh into the range from minus one to one. That squeeze keeps the numbers in range. In a vanilla RNN, the plain ungated kind, repeated squashing and multiplying by U make old inputs fade step by step. Here is one step as dataflow:
That washout is exactly what the gates were invented to stop (Gated RNNs: GRU and LSTM).
Each hand-off fades the past a little, so over a long sequence the early words wash out.
Figure 3. A voice command, fade exaggerated (illustrative).
This section builds on the backward pass in Backpropagation and Gradient Checking, where the chain rule carries a gradient back through layers, and on Cross-Entropy Loss.
In an RNN, gradients flow back through time as well as depth: backpropagation through time (BPTT). Unrolled for T steps, the RNN is a depth-T network that reuses the same weights at every layer, with a loss Lt at every step (say cross-entropy, the standard loss for picking the right class). The symbol ∂ marks a partial derivative: how much one quantity moves when another moves.
Take the voice command of Figure 3. The loss at "on" reaches the memory of "turn" through two hand-offs, so its gradient carries two factors, one per step back. The third line of the card writes that product for a sequence of any length.
Here each factor is one number w. In a real RNN each factor is a Jacobian (a matrix of partial derivatives), but the exponential dependence on T is the same.
The exploded end has a standard fix, clipping (Section 2.2).
Figure 5 calls a gradient usable while step 1 still gets between 10−2 and 102 times its size at the loss; below the band it has vanished, and above the band it has exploded. The edges are a rule of thumb for this chart (illustrative).
Figure 5. Gradient size wt on a log scale, from the loss back to step 1.
About 6% reaches step 1, still usable; longer inputs need gated RNNs.
Exploding gradients have a cheap fix, gradient clipping: if the gradient is longer than a threshold τ, scale it down to that length and keep its direction (Pascanu et al., 2013).
In a speech or language RNN, one long batch can push the factors above 1. Take w = 1.5 over 28 steps (Figure 5, illustrative): by the gradient-size card above, step 1 gets about 6 × 104 times the gradient, and one update that large throws the weights far off. With τ = 1, clipping scales that gradient back to length 1 and keeps its direction.
The section Gradient clipping of Optimizers in Practice: AdamW, SAM, Schedules and Clipping shows where the clip runs in the training step and how to pick τ.
Clipping tames the explosions. The bigger problem is vanishing gradients: with no control over what passes forward, old information washes out and the RNN forgets context.
In the truncated BPTT code, rnn is the layer from Section 1, head a linear output layer, loss_fn a per-step loss, optimizer covers both, x and y are inputs and per-step targets with time on axis 1, and k is the chunk length. A longer k learns longer dependencies but stores more steps; a shorter k is cheaper, but no gradient reaches back more than k steps.
h = None
# k steps per chunk; carry h
xs = x.split(k, 1)
ys = y.split(k, 1)
for x_c, y_c in zip(xs, ys):
out, h = rnn(x_c, h)
pred = head(out)
loss = loss_fn(pred, y_c)
# backprop this chunk only
optimizer.zero_grad()
loss.backward()
optimizer.step()
# keep values, cut the graph
h = h.detach()
The same chunking works unchanged for nn.GRU; for nn.LSTM, detach both parts of its (h, c) state, as in Gated RNNs: GRU and LSTM.
Clipping fixes gradients that explode, but how does a network keep information for hundreds of steps when gradients vanish? That is the question Gated RNNs: GRU and LSTM answers.