← All tutorials

MGDA (Multiple Gradient Descent Algorithm)

MGDA picks one weight per task, each zero or more and all summing to one, so the weighted sum of the task gradients is as short as possible, then steps against that sum. For a small step every task's batch loss falls, unless that sum is zero, which makes the model Pareto-stationary (Section 5).

On this page

A multitask network has one shared encoder, with shared weights θ_sh, and one small head per task (Figure 1). The encoder turns the input into a representation Z that every head reads. Each task's gradient on the shared weights points where its loss rises fastest. Désidéri introduced MGDA in a 2009 report; Sener and Koltun (2018) made it practical for deep networks.

1 Dominated and Pareto-optimal models

• Model B is dominated when some model A is no worse on any task and strictly better on at least one.
• A model is Pareto-optimal when no task can improve without another getting worse.
• The Pareto front is the curve of Pareto-optimal models' task losses.

Take NYUv2, indoor photos, with one network learning semantic segmentation (seg, a class for every pixel) and depth (metres to every pixel). Three checkpoints have illustrative (seg, depth) losses A = (0.9, 0.5), B = (0.9, 0.7) and C = (0.8, 0.9). A dominates B: the same seg loss and a lower depth loss. A and C each win one task, a trade-off. In Figure 2, B plays the part of ②, straight above A, and A the part of ③.

MGDA looks for steps that make no task worse, so it moves a dominated model toward the front.

2 The average step can hurt one task

With two shared weights, each task gradient is an arrow: g₁ = (2, 0) and g₂ = (−1, 0.5), the pair from the intro's average-step example. Training steps against a direction d built from them.

On NYUv2 with illustrative gradients g_seg = (2, 1) and g_depth = (−6, 2), the average is (−2, 1.5). Its dot product with g_seg is −2.5 and with g_depth 15, so the average step raises seg’s loss. Figure 3 after the card shows the same failure on the pair above: the average is 127° from g₂, so ⟨g₂, d⟩ = −0.38 and task 2’s loss rises.

3 The minimum-norm point

A convex combination of the T task gradients is a weighted sum whose weights w_i are zero or more and sum to one. The allowed weight lists form the simplex Δ, and all convex combinations together form the convex hull of the gradients.

On the NYUv2 pair the shortest point of the segment is d* = (0.154, 1.231), at w_seg = 0.769, with squared length 1.538 (illustrative). Figure 4 draws the same idea on the pair of Section 2, where d* sits at w₁ = 0.35 with both scores 0.11.

On the NYUv2 pair, ⟨g_seg, d*⟩ = ⟨g_depth, d*⟩ = 1.538, equal to ‖d*‖² (illustrative). On Figure 4’s pair both scores read 0.11 at d*.

3.1 The two-task closed form

On the NYUv2 pair with seg as task 1: g_depth − g_seg = (−8, 1), its dot product with g_depth is 48 + 2 = 50, and ‖g_seg − g_depth‖² = 64 + 1 = 65, so w_seg = 50 over 65 = 0.769 and w_depth = 0.231 (illustrative). On Figure 4’s pair the same formula gives 3.25 over 9.25 = 0.351, the tool’s 0.35.

4 The minimum-norm point shrinks as training goes on

5 Pareto-stationary points

A point is Pareto-stationary when d* = 0, so the gradients show no direction that lowers every loss.

Add a third NYUv2 task, surface normals (the direction each pixel’s surface faces), with illustrative g_normals = (1, −2). Then 0.4 (2, 1) + 0.2 (−6, 2) + 0.4 (1, −2) = (0, 0), so this point is Pareto-stationary. Figure 6 draws the same case on its own three gradients. The NYUv2 pair alone is the other case: its shortest blend has length 1.240, never 0, and both tasks gain 1.538 = ‖d*‖².

A Pareto-stationary point need not be Pareto-optimal. The gradients can also cancel at a saddle point, where losses rise in some directions and fall in others.

6 Frank-Wolfe for three or more tasks

With three or more tasks there is no closed form, so the 2018 paper uses Frank-Wolfe, an iterative method.

On the three NYUv2 gradients of Section 5, from one third each the blend is (−1, 0.333). Seg and normals tie for the smallest dot product, −1.667, and the tie goes to the first task, seg. The corners then run seg, normals, depth, normals, depth, normals and onward. After 22 moves the best move size γ falls below 0.00001, the TOL of the code in Section 8.1, so iteration 23 stops, at weights 0.4, 0.2 and 0.4 and blend (0, 0) (illustrative gradients). Figure 7 draws the same moves on its own three gradients.

The official code (min_norm_solvers.py) runs gradient descent on the weights instead and pulls them back onto the simplex after each step, starting from the best pair of tasks. Its loop counter never increases, so its cap of 250 never fires; it stops when the weights change by less than 1e-5, summed over all tasks.

7 When MGDA runs and what it costs

• MGDA solves the weights afresh on every batch from its mean losses and keeps no state, so they can change from batch to batch.
• Only the gradients on the shared weights enter G (MGDA-UB in Section 7.1 uses Z). G is only T by T, so the solve takes negligible time, per the 2018 paper. Memory goes to the T by P task-gradient matrix, where P, the shared-weight count, is often in the millions.
• In the official code each head's gradient is multiplied by its task weight; the 2018 paper's pseudocode leaves it unscaled (how each head's gradient is scaled).
• In the official code each step needs one extra forward and one extra backward pass per task (Figure 8).

7.1 MGDA-UB, gradients at the representation

MGDA-UB (UB for upper bound) takes each task's gradient at Z (Figure 1) instead of at θ_sh.

On NYUv2 with a shared encoder output Z, plain MGDA runs the encoder backward once per task to get each g_i (Figure 8); MGDA-UB needs one backward pass through each head, none through the encoder, and then one encoder pass for the chosen blend.

The 2018 paper claims MGDA-UB still lowers every loss or stops at a Pareto-stationary point when the Jacobian of Z has full rank, meaning the weights can move Z in every direction. This does not hold in general. In a two-weight example built for this page, with a full-rank 2 by 2 Jacobian, MGDA-UB picks 0.5 and 0.5, and a small step against that direction raises task 1's loss; exact MGDA picks 0.7 and 0.3.

The 2018 paper times both on Cityscapes (semantic segmentation, instance segmentation and depth) and CelebA (200K face images, 40 yes/no attributes as tasks, ResNet-18).

8 MGDA in a PyTorch training step

MGDA sits between the losses and the optimizer step (Figure 10).

8.1 The minimum-norm weight solver

min_norm turns G into the weights by the Frank-Wolfe iteration of Section 6.

import torch

ITERS, TOL = 250, 1e-5

def min_norm(G):
    # G[i, j] = g_i . g_j, (T, T)
    T = len(G)
    w = G.new_ones(T) / T
    for _ in range(ITERS):
        t = (G @ w).argmin()
        dd = w @ G @ w
        dg = w @ G[t]
        gg = G[t, t]
        den = dd - 2 * dg + gg
        if den <= 0:
            break
        gam = (dd - dg) / den
        gam = gam.clamp(0, 1)
        if gam < TOL:
            break
        w = (1 - gam) * w
        w[t] += gam
    return w

1. 1The iteration cap and the tolerance on gam (Section 9).
2. 2Equal weights 1/T, marker ① in Figure 7.
3. 3Picks task t, the one the combination agrees with least (Section 6 card, line 1).
4. 4dd, dg and gg are dot products of the combination and g_t, den is their squared distance (the loop stops at zero), and gam is the Section 3.1 formula.
5. 5Stops when gam is nearly zero, otherwise moves toward task t (Section 6 card, lines 2 and 3).

8.2 The full training step

mgda_step runs one training step for two regression tasks with one shared forward; paste it below the min_norm solver of Section 8.1.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import grad
from torch.optim import Adam

trunk = nn.Linear(16, 32)
h1 = nn.Linear(32, 1)
h2 = nn.Linear(32, 1)
heads = nn.ModuleList([h1, h2])
theta = list(trunk.parameters())
H = list(heads.parameters())
opt = Adam(theta + H, lr=5e-4)
keep = dict(retain_graph=True)

def mgda_step(x, ys):
    # x: (B, 16), ys: two (B, 1)
    z = F.relu(trunk(x))
    losses = []
    for h, y in zip(heads, ys):
        L = F.mse_loss(h(z), y)
        losses.append(L)
    rows = []
    for L in losses:
        g = grad(L, theta, **keep)
        v = [u.ravel() for u in g]
        rows.append(torch.cat(v))
    # Gm: (T, P), one row per task
    Gm = torch.stack(rows)
    w = min_norm(Gm @ Gm.T)
    opt.zero_grad()
    Ls = torch.stack(losses)
    (w @ Ls).backward()
    opt.step()
    return w

1. 1A trunk, two heads, and theta, the shared weights. keep is retain_graph=True since the final backward reuses the graph.
2. 2One forward and each task's batch-mean loss.
3. 3One gradient per task on theta, flattened to a row of P numbers (box 1 in Figure 1). The training-loop page shows how each task gradient is computed.
4. 4The Gram matrix and the solve of Section 8.1, box 2 in Figure 1.
5. 5One backward of the weighted loss. theta receives exactly d* and each head its gradient times w_i, boxes 3 and 4 in Figure 1.

• For MGDA-UB, pass [z] instead of theta to grad, so each call goes back only through its head, and z is far smaller than the shared weights.
• The official full-MGDA path runs one forward per task, so dropout draws new random masks and batch norm updates its statistics per task, unlike this snippet's one forward.

9 MGDA's settings

MGDA has no hand-set loss weights, but four settings change its step.

Gradient normalization

Without a normalizer n_i, a longer gradient gets less weight, and a loss scaled by 10 has a 10 times longer gradient (see how loss scale enters the gradient). The official sample config uses loss+; the 2018 paper uses none.

With n_i = ‖g_i‖, the NYUv2 pair becomes unit arrows (0.894, 0.447) and (−0.949, 0.316). Their shortest blend has weights 0.5 and 0.5, so the step on the raw gradients is the plain average (−2, 1.5) again and seg’s score falls back to −2.5, where the raw pair gave w_seg = 0.769 and 1.538 for both (illustrative). Figure 11 shows the same switch on Figure 4’s pair.

Before solving, each gradient can be divided by a normalizer so one big task does not decide the blend alone.

g_i ← g_in_i

Like MGDA-UB in the eq card in Section 7.1, this changes what the solve sees; here it moves the weight of the eq card in Section 3.1, 0.769 raw against 0.5 after l2. The weights found this way then multiply the raw losses.

n_i = 1, ‖g_i‖, L_i, or L_i‖g_i‖

none, l2, loss and loss+. l2 keeps only directions, so with two tasks it always gives 0.5 and 0.5, the plain average (Figure 11). loss and loss+ make a small-loss task's row longer, so it gets less weight.

So the normalizer decides how much each task’s size counts in the blend.

Solver stopping

The Section 8.1 snippet stops after ITERS (250) iterations or once gam is below TOL (1e-5). A higher cap or lower tolerance gives a combination nearer d* at more cost; the opposite is faster and leaves it longer. A cap of 1 makes one Frank-Wolfe move from equal weights. In 100 random cases of four tasks, each gradient 6 random normal numbers, the snippet hit the cap in 77, leaving the combination a median 0.18% and at most 4.8% too long. The official solver stops on the weight change instead (Section 6).

MGDA-UB on or off

On (the sample config) takes gradients at Z: one encoder backward instead of T, but an approximation (Section 7.1). Off solves the exact problem.

Learning rate

The guarantee needs a small step. A larger rate trains faster but can overshoot and raise a task's loss; a smaller one keeps every loss falling, more slowly. The sample config uses Adam at 5e-4; the training script cuts it by 15% every 10 epochs.

10 Which stationary point MGDA reaches

Which Pareto-stationary point MGDA reaches depends mainly on the network's initialization. Pareto MTL (Lin et al., 2019) gives each run a preference direction limiting which part of the front it may end in, so several runs spread along the front, as sketched below.

11 MGDA among the balancing methods

MGDA solves its weights fresh each batch from the task gradients' lengths and angles. GradNorm instead learns per-task loss weights so tasks train at similar rates.

12 References

1. Désidéri, J.-A. (2009). Multiple-Gradient Descent Algorithm (MGDA). INRIA Research Report RR-6953.
2. Désidéri, J.-A. (2012). Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathématique, 350(5-6), 313-318.
3. Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012). Indoor Segmentation and Support Inference from RGBD Images. ECCV.
4. Sener, O., and Koltun, V. (2018). Multi-Task Learning as Multi-Objective Optimization. NeurIPS. arXiv:1810.04650.
5. Lin, X., et al. (2019). Pareto multi-task learning. NeurIPS.
6. Official code at isl-org/MultiObjectiveOptimization (discontinued), and the LibMTL library.