← All tutorials

Cross-Entropy Loss

Cross-entropy scores a predicted probability against the true label: confident and right costs almost nothing, confident and wrong costs a lot. The labels decide which form to use, and KL Divergence picks up when the target is itself a set of chances.

On this page

1 Binary cross-entropy (BCE)

BCE scores a yes-or-no prediction. The model says how likely a class is present, and BCE measures how wrong that probability was.

In RLHF (reinforcement learning from human feedback), a reward model learns which of two answers people preferred, and this card is its loss (Ouyang et al. 2022). The gap is its score for the first answer minus its score for the second. A sigmoid, an S-shaped curve that turns any score into a number between 0 and 1 (0 becomes 0.5, large positive scores land near 1, large negative ones near 0; its equation is in Section 3), turns the gap into a probability p, and label 1 means people preferred the first answer.

Each class is its own yes-or-no bet, so BCE handles classes that co-occur. The target is a multi-hot vector, one 0 or 1 per class, a 1 for every class present. Sigmoid plus BCE is exactly the loss logistic regression trains on.

import torch

def bce_with_logits_loss(
        logits, y):
    # logits, y: (N, C);
    # y is multi-hot 0 or 1
    p = torch.sigmoid(logits)
    lp = torch.log(p)
    lq = torch.log(1 - p)
    ll = y * lp + (1 - y) * lq
    return -ll.mean()

1. 1Squashes each logit into p, the sigmoid, eq card in Section 3, line 1.
2. 2The two charges, eq card in Section 1, lines 1 and 2.
3. 3Keeps the log that matches each label, the bracket of the eq card in Section 1, line 3.
4. 4Flips the sign and averages over examples and classes, eq card in Section 1, lines 3 and 4.

PyTorch's nn.BCEWithLogitsLoss computes the same from the logits in a numerically safer form.

2 Categorical cross-entropy

Categorical cross-entropy (CE) keeps only BCE's yes term, minus the log of the probability, for the one true class, because exactly one class can be true. A softmax exponentiates each logit and divides by the sum of those exponentials, so the C classes get positive chances that share one total of 1 (its equation is in Section 3).

The label can be a one-hot vector (all zeros except a single 1) or just the index of that 1. PyTorch takes both:

import torch
from torch import nn

# the teacher's logits for
# cat, dog, car, bird
logits = torch.tensor(
    [[4.6, 2.2, 0.8, 0.0]])
# BCE: tags cat and car
multi_hot = torch.tensor(
    [[1., 0., 1., 0.]])
# CE: the one class, cat
index = torch.tensor([0])

bce = nn.BCEWithLogitsLoss()(
    logits, multi_hot)
ce = nn.CrossEntropyLoss()(
    logits, index)

1. 1One example's logits, the table in Section 3.
2. 2The two target shapes, the two rows of Figure 1.
3. 3Computes both losses from the logits; bce averages four bets to 0.84, ce is 0.12, minus the log of cat's 0.890.

CE is the negative log likelihood of the true class: lower loss means higher probability. Maximum Likelihood and EM, section 6 derives it. CTC loss uses the same negative log over whole sequences, and contrastive learning's InfoNCE loss is categorical cross-entropy with the positive pair as the correct class (Contrastive Learning).

import torch

def cross_entropy_loss(
        logits, target):
    # logits: (N, C)
    # target: (N,) class ids
    probs = torch.softmax(
        logits, dim=1)
    n = len(target)
    rows = torch.arange(n)
    p_true = probs[rows, target]
    nll = -torch.log(p_true)
    return nll.mean()

1. 1Turns each row of logits into chances that add to 1, the softmax, eq card in Section 3, line 2.
2. 2Picks each example's chance of its correct class, p_y in the eq card in Section 2.
3. 3Charges minus its log, eq card in Section 2, line 1, then averages over the batch.

3 Softmax competes, sigmoid does not

Softmax is a competition with a fixed prize: one class's probability grows only by taking probability from the others. Sigmoid is the opposite: the same logits become separate yes-or-no bets, the BCE case for labels that co-occur.

Use softmax when exactly one label is right, sigmoid when labels can co-occur. Pick the wrong one and the loss teaches the wrong lesson.

The teacher on the Knowledge Distillation page, a large trained model whose chances a small student model learns to copy, scores a cat photo for cat, dog, car and bird with illustrative logits, and the softmax turns them into chances.

With label cat, the loss of Section 2 is 0.12 for the teacher's 0.890 and 0.80 for an illustrative weaker model that puts 0.451 on cat.

4 Why classification uses cross-entropy, not MSE

Why not mean squared error (MSE), the average squared gap, for classification? Take one example with true label 1 and predicted probability p. (Regression uses squared error because its targets are numbers, not classes.)

Turned the wrong way round, the reward pair of Section 1 gives the preferred answer a probability of 0.119.

5 The scoring pipeline, end to end

Follow the reward pair of Section 1 from logit to loss. Its gap of +2.0 is the logit, the sigmoid of Section 3 turns it into 0.881, and the BCE line charges 0.881 against label 1 for a loss of 0.13.

6 KL divergence

When the target is itself a set of chances, such as a distillation teacher's output or smoothed labels, cross-entropy splits into two parts.

7 Cross-entropy variants

Plain cross-entropy assumes the labels are exactly right and every class matters equally. Three variants relax that.

• Label smoothing mixes the one-hot target with a uniform spread over all classes (0.1 of the mass in Szegedy et al. 2016), so the model is never rewarded for total certainty; Knowledge Distillation trains on soft targets the same way. Smoothed labels are a soft target, so the loss is the target's own spread plus KL divergence and never reaches 0 (KL Divergence).
• Focal loss scales down easy examples so training spends its effort on the hard ones (Lin et al. 2017 found a focusing strength γ of 2 worked best); Handling Class Imbalance has its card.
• Class weighting multiplies each class's loss by its own weight, a cheap fix for imbalance.

Each is one argument or one call in PyTorch and torchvision:

import torch
import torch.nn.functional as F
from torchvision.ops import (
    sigmoid_focal_loss)
# logits, index and multi_hot
# as in the nn listing above
smooth = F.cross_entropy(
    logits, index,
    label_smoothing=0.1)
focal = sigmoid_focal_loss(
    logits, multi_hot,
    gamma=2.0, reduction="mean")
# illustrative, one per class
class_weights = torch.tensor(
    [1., 2., 2., 4.])
weighted = F.cross_entropy(
    logits, index,
    weight=class_weights)

1. 1Label smoothing, 0.1 of the target spread over all classes.
2. 2Focal loss on the multi-hot target, focusing strength 2.
3. 3Class weighting, a weight per class multiplying its loss.

One prediction, scored against one label. When the target is a set of chances, the next page, KL Divergence, takes over.

8 References

1. Lin, T.-Y., et al. (2017). Focal Loss for Dense Object Detection. ICCV. arXiv:1708.02002.
2. Ouyang, L., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS. arXiv:2203.02155.
3. Szegedy, C., et al. (2016). Rethinking the Inception Architecture for Computer Vision. CVPR. arXiv:1512.00567.