On this page
A multilayer perceptron (MLP) stacks linear layers with a nonlinear activation function between them, so it can draw boundaries that one straight line cannot; training tunes every weight to lower the loss.
An MLP takes one gradient step per batch over many passes through the data: a forward pass predicts, and a backward pass, about twice the forward work, finds each weight's gradient.
Logistic regression, the straight-line classifier, fails on the XOR layout: four blobs at the corners of a square, each class on one diagonal.
Every model on this page trains on these blobs: four Gaussian clouds, 1,200 points, 8% of the labels flipped.
The simplest nonlinear function is the ReLU, max(0, x), also called a hinge function: flat, then a straight ramp, with one kink. A sum of shifted hinges is a piecewise linear function that can follow a curve.
The hinge fit in NumPy.
import numpy as np
K = 8
x = np.linspace(-2, 2, 200)
y = x ** 2
j = np.arange(K)
t = -2 + 4 * (j + 0.5) / K # t_k
D = x[:, None] - t
ramps = np.maximum(0, D)
ones = np.ones((len(x), 1))
Phi = np.hstack([ones, ramps])
ls = np.linalg.lstsq
sol = ls(Phi, y, rcond=None)
coef = sol[0] # [a, w_1..w_K]
fit = Phi @ coefFigure 5. Each hinge adds one kink; with 8 the fit follows the parabola closely except at the left edge.
A one-hidden-layer MLP is a sum of hinges whose positions and weights are learned. Each unit (the name comes from Rosenblatt's perceptron) takes a weighted sum of its inputs, adds a bias, and applies the activation.
In PyTorch the first Linear makes H lines from the 2 inputs, ReLU kinks each, and the last Linear mixes them into one score.
import torch.nn as nn
H = 128 # hidden units
mlp = nn.Sequential(
nn.Linear(2, H),
nn.ReLU(),
nn.Linear(H, 1),
)Without the ReLU, the two layers collapse.
A forward pass turns one input point into a probability: each hidden unit computes a weighted sum, ReLU sets negative sums to zero, and the output unit adds the results and applies the sigmoid.
Figure 6. With hand-picked weights h1 fires on the top-left blob and h2 on the bottom-right.
ReLU sets negative sums to 0σ the sigmoid, maps the sum to a probabilityThe same forward pass in PyTorch; no_grad marks it as prediction only, with no gradients kept.
import torch
# x: (2,); W1: (2, 2); W2: (1, 2)
with torch.no_grad():
h = torch.relu(W1 @ x + b1)
s = W2 @ h + b2
y_hat = torch.sigmoid(s)Training needs every weight's gradient. Backpropagation applies the chain rule layer by layer, from the output back to the input, and reuses each layer's result for the layer before it. For one first-layer weight, with L the binary cross-entropy loss of Logistic Regression.
In PyTorch loss.backward() computes all of these at once (autograd). The backprop_pytorch notebook in the Machine-Learning-Concepts repo writes the backward pass by hand.
Capacity is how complicated a boundary the model can draw; width (units per layer) and depth (number of layers) both add to it. A model with higher training and lower test accuracy is overfitting.
Here the large model overfits only when nothing penalizes its weights.
Figure 8. At capacity 3 the boundary bends around mislabeled points; at 4 the L2 penalty smooths them out.
The large model's boundary bends around single points because its weights grew large to fit every flipped label. Regularization adds a penalty on the weights to the training loss, in two common forms.
L1 pushes many weights to exactly zero, giving a sparse model where only a few weights do the work.
Dropout sets a random share of units to zero at each training step, so the net cannot rely on any one unit; see Regularization.
ReLU is the default activation for hidden layers because it is cheap and trains well; other activations suit other jobs.
Output layers use their own: sigmoid for yes-or-no, softmax (class scores into probabilities that add up to 1) for many classes.
Each activation is one PyTorch module.
import torch.nn as nn
hidden = nn.ReLU()
leaky = nn.LeakyReLU(0.1)
yes_no = nn.Sigmoid()
zero_centered = nn.Tanh()
many_classes = nn.Softmax(dim=-1)
gelu = nn.GELU() # transformersFigure 10. Each activation 1 to 4 has a kink or a curve, which lets a network bend its boundary.
One training step; the loss takes raw scores (logits) and applies the sigmoid inside.
import torch
from torch import nn
from torch.optim import AdamW
model = nn.Sequential(
nn.Linear(2, 64),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(64, 1),
)
params = model.parameters()
opt = AdamW(params, lr=1e-3)
loss_fn = nn.BCEWithLogitsLoss()
x = torch.randn(128, 2)
y = torch.randint(0, 2, (128, 1))
y = y.float()
model.train() # dropout on
loss = loss_fn(model(x), y)
opt.zero_grad()
loss.backward()
opt.step()The step runs once per batch, for as many epochs as the validation score improves; memory holds the weights, the activations kept for the backward pass, and the optimizer's two running averages per weight.
| Setting | Where it enters | Raise it | Lower it | Default |
|---|---|---|---|---|
| Hidden units | Width of W1 | More kinks; can overfit | Fewer; at 0 no hidden layer | 100 (MLPClassifier) |
lr | Step size of each update | Faster; too high diverges | Slower | 1e-3 (AdamW) |
weight_ | Shrinks every weight each step | Smoother boundary; too high underfits | No shrinking at 0 | 1e-2 (AdamW) |
Dropout(p) | Share of hidden units zeroed each training step | Stronger regularization, slower fit; near 1 drops almost every unit | Weaker; 0 turns it off | 0.5 (nn.Dropout) |
alpha | MLPClassifier's L2 λ | As λ above | 1e-4 | |
Call model.eval() and wrap prediction in torch.no_grad(), or dropout stays on and memory grows. Use BCEWithLogitsLoss on raw scores rather than a sigmoid then BCELoss, which PyTorch documents as less numerically stable. AdamW's weight decay shrinks weights directly instead of adding the L2 term to the loss; with Adam the two differ.
| Cause | Symptom | Fix |
|---|---|---|
| Dead ReLUs | units output zero for every input, so they stop learning and the loss stalls | Leaky ReLU, or a smaller learning rate lr (the step size of each update) |
| Bad init scale | activations blow up or fade layer by layer | He init for ReLU layers, starting weights scaled to each layer's input count so signals keep their size (nn.init.) |
| Learning rate too high | the loss jumps around or turns NaN | lower lr; see Optimizers from SGD to Adam |
MLPs treat every input feature alike; CNN Basics: Convolution, Stride and Pooling shows how convolutions share weights across an image to see local patterns.