Gradient Descent

1 The loss landscape

The loss, one number for how wrong the model is, forms the loss landscape over all the weights; the gradient is its slope at the current weights, pointing where the loss rises fastest. Gradient descent steps the weights against the gradient until it is near zero, leaving weights with low training loss.

A local minimum is lowest only nearby, the global minimum lowest everywhere. Gradient descent cannot tell them apart; it moves downhill from wherever it starts.

Figure 1. The start decides which side of the peak gradient descent goes down.

1.1 The gradient descent update

Each step costs one forward and backward pass over the whole training set and carries only the weights θ to the next. One random minibatch per step instead is stochastic gradient descent (SGD), on SGD, Momentum and Nesterov.

Each step moves the weights a little way downhill on the loss.

θ ← θ − α·∇L(θ)

The update replaces the current weights θ with θ minus a step. The step points against the gradient ∇L(θ), because the gradient points where the loss rises fastest; the minus sign makes the loss go down. Its length is the learning rate α times the gradient: where the loss is steep the steps are long, and near a minimum the gradient shrinks, so the steps shrink on their own. A larger α lowers the loss in fewer steps but can overshoot the minimum; at α = 0 nothing moves, and past a limit set by how sharply the loss curves every step overshoots further and the weights diverge. PyTorch's SGD defaults to α = 0.001.

The update on the toy loss, the sum of θ²; the loop stops once the slope is shorter than tol. A smaller tol stops later, closer to the minimum; at 0 it runs all 1000 steps.

import numpy as np
from numpy.linalg import norm
def grad_L(theta):
    return 2 * theta  # sum θ²
theta = np.array([3.0, -2.0])
alpha, tol = 0.1, 1e-6
for t in range(1000):
    grad = grad_L(theta)
    if norm(grad) < tol:
        break  # flat: done
    move = alpha * grad
    theta = theta - move

1The slope of the toy loss, 2θ.
2Start θ, rate α, tolerance.
3Stops once the slope is flat.
4The update equation.

Figure 2. Steps shrink as the slope flattens.

θ ← θ − α·∇L(θ)

The red dot is the current weight θ. The purple tangent shows the slope ∇L(θ), the straight line that tilts the way the curve tilts there. The steeper the tangent, the longer the red arrow, the step α times the slope, and the arrow always points downhill.

θ the current weight (red dot); α·∇L(θ) the step (red arrow)∇L(θ) the slope (purple tangent)α the learning rate← is replaced by

1.2 Starting point and learning rate

The start picks the minimum and the learning rate decides whether it arrives: a small rate crawls, a medium one converges, a larger one oscillates across the minimum and one too large diverges. Ending in the local minimum is optimization error: better weights exist and training missed them.

Start Learning rate 0.051

gradient descent trail current weights

Figure 3. One run at the slider's rate; the ring marks the minimum it reaches.

2 One gradient step in PyTorch

torch.optim.SGD without momentum is exactly the update; the snippet fits a 3-weight linear model to 512 points with α = 0.1.

import torch
X = torch.randn(512, 3)
y = X @ torch.randn(3, 1)
net = torch.nn.Linear(3, 1)
ps = net.parameters()
opt = torch.optim.SGD(ps, 0.1)
for step in range(100):
    opt.zero_grad()
    err = net(X) - y
    loss = (err ** 2).mean()
    loss.backward()  # grads
    opt.step()  # θ ← θ − α·g

1512 points from three random true weights.
2Linear model and SGD with α = 0.1.
3Clear, mean squared error, backward pass.
4The update equation.

Each step is a forward and backward pass over all 512 points, so time grows with the data and memory holds every point's activations; plain SGD keeps no other state. The rate has no universal value; the torch.optim documentation example uses 0.01.

Gradients accumulate, so a missing opt.zero_grad() adds every old gradient into each new step.
A summed loss instead of a mean multiplies the gradient, and the step, by the number of points.
A hand-written w -= lr * w.grad must run inside torch.no_grad(), or autograd records it.

3 Other optimizers

With one rate for every weight and only the current gradient, plain gradient descent oscillates across steep directions and crawls along gentle ones; momentum fixes this on SGD, Momentum and Nesterov, and per-weight step sizes and Adam on Adaptive Optimizers: AdaGrad, RMSProp and Adam. Weight decay, learning rate schedules and gradient clipping around the step are on Optimizers in Practice: AdamW, SAM, Schedules and Clipping.

The usual classifier loss is Cross-Entropy Loss.

4 References

Cauchy, A. (1847). Méthode générale pour la résolution des systèmes d'équations simultanées. Comptes Rendus de l'Académie des Sciences. (gradient descent itself)