Diagnose which failure you have, batch every task in every step, admit only the losses that must shape the shared trunk, and know what no optimizer can fix.
Figure 1. Checks 1 to 3, the failure each reveals, and the methods that fix it.
2 Measuring each task's loss and gradient
Each check reads, per task i, its loss on its own batch and its gradient gi over the shared trunk θsh: one vector over all shared weights, heads left out.
Figure 2.1 Each loss gives 2 one gradient over the trunk; 3 heads stay out.
As a running example, take NYUv2, indoor photos with one network learning depth (meters to every pixel) and semantic segmentation (seg, a class for every pixel). Illustrative gradients on two shared weights are gdepth = (−6, 2) and gseg = (2, 1). Task 1 is depth and task 2 is seg everywhere on this page, in the figures and in the code: task 1 is the long, fast task and task 2 the one that stalls or gets hurt.
Measure each task on its own, its average loss and its gradient on the shared weights, before choosing any fix.
Li = 1Bi Σnℓi(xn)
Task i's loss Li averages its per-example loss ℓi over the Bi examples xn of that task in the batch; Σ adds over n from 1 to Bi. Averaging keeps the loss on one scale whatever the batch size.
gi = ∇θshLi
∇ with subscript θsh takes the gradient with respect to the shared weights only: gi has one entry per shared weight, how fast Li rises as that weight grows. Section 4 compares the lengths of these vectors, Section 5 the angle between them.
So every later check on this page reads these two measured numbers per task: here depth’s and seg’s batch-mean losses and their gradients (−6, 2) and (2, 1) on two shared weights.
Look at task_grad, which turns one task's batch-mean loss into one vector over the trunk.
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import grad
torch.manual_seed(0)
trunk = nn.Linear(16, 32)
head1 = nn.Linear(32, 1)
head2 = nn.Linear(32, 1)
shared = list(trunk.parameters())
# one batch per task, own sizes
x1 = torch.randn(64, 16)
# task 1: targets 100x larger
y1 = 100 * torch.randn(64, 1)
x2 = torch.randn(16, 16)
y2 = torch.randn(16, 1)
def task_grad(head, x, y):
out = head(trunk(x))
loss = F.mse_loss(out, y)
gs = grad(loss, shared)
# flatten, join: one vector
v = [g.flatten() for g in gs]
return loss, torch.cat(v)
L1, g1 = task_grad(head1, x1, y1)
L2, g2 = task_grad(head2, x2, y2)
# g1 has 544 entries
1shared holds the trunk's weights only, marker 3 in Figure 2.
2One batch per task, 64 and 16 examples, marker 1.
Log each raw loss and gradient, before task weights and before the mixed-precision loss scaler multiplies them (what to log in the training loop).
PCGrad, GradVac, CAGrad and MGDA (Sener and Koltun, 2018) combine task gradients, so they compute every gi each step anyway. With plain weighting or Uncertainty Weighting each logged step costs one backward pass per task; every few hundred steps is a rule of thumb. On a large model, log at the last shared layer only, for every task, as GradNorm does (Chen and colleagues, 2018) for about 5% extra training time on NYUv2.
3 Loss ratio shows learning speed
With fixed weights a task's share of each step follows its gradient, not how far it still has to go, so a lagging task gets no extra push.
In the failing run of Figure 3, whose logs are illustrative, task 1’s ratio falls to 0.1 while task 2’s stalls near 0.86; with task 2 as seg, seg is the task barely learning.
The loss ratio says how much of its starting loss a task still has, so a slow task stands out.
L̃i(t) = Li(t)Li(0)
Li is the batch-averaged loss of Section 2. The loss ratio L̃i divides task i's loss at step t by its loss at step 0, so every task starts at 1 whatever its units, and a faster fall means faster learning. For a cross-entropy task whose first loss depends on the random start, log C, the loss of a uniform guess over C classes, can stand in for Li(0). Single-batch losses are noisy, so compare ratios averaged over many steps; act when one stays over twice another and the gap keeps growing, a rule of thumb.
So a ratio that stays near 1 flags a task that is not learning: in the failing run seg’s ratio stalls near 0.86 while depth’s falls to 0.1.
Look at ratio1, task 1's loss ratio.
# L1, L2 from task_grad
# at step 0
L0 = [L1.item(), L2.item()]
# at each logged step
ratio1 = L1.item() / L0[0]
ratio2 = L2.item() / L0[1]
Figure 3.1 Healthy ratios fall together; 2 failing ones split as 3 task 2 stalls.
The step follows the sum of the task gradients, so the longest sets its direction.
In the running example, task 1 is depth and task 2 is seg, so ρ = 6.325 over 2.236 = 2.828: depth’s push is almost three times seg’s (illustrative pair). A common rule of thumb flags a scale problem only when norms sit more than 10 times apart, so this pair has no loss-scale problem; its harm comes from size and direction together, which Section 5.2 tests.
The ratio of the two gradient lengths says whether one task’s loss scale is drowning the other.
ρ = ‖g1‖‖g2‖
g1 and g2 are the gradients of Section 2, and a short one can explain a slow loss ratio from Section 3. The norm ‖gi‖ is the vector's length, the square root of the sum of its squared entries. The ratio ρ (rho) says how many times longer task 1's gradient is: near 1 the two pull about equally. Multiplying a loss by a constant multiplies its gradient's norm by the same constant, so the ratio exposes a scale gap. Centimeters instead of meters multiply a squared loss, and its gradient, by 10,000. Norms over 10 times apart, averaged over many steps, mean a scale problem, a rule of thumb.
So a ρ far from 1 is a loss-scale problem that loss weights can fix; depth and seg, at 2.828, are not that far apart.
When two task gradients point partly against each other, a step that lowers one loss raises the other. The cosine measures this regardless of length.
The log keeps m, a running average of the cosine over earlier steps; say it has settled at 0.30 (illustrative). In the running example this step’s cosine is −0.707, which moves m only to 0.98 × 0.30 + 0.02 × (−0.707) = 0.280: one conflicting batch barely shifts it, and many in a row drag it below 0.
The cosine says whether the two tasks pull the same way, and a running average stops one noisy batch from deciding.
cos ψ = ⟨g1, g2⟩‖g1‖ ‖g2‖
This is the direction half of the pair whose lengths Section 4 compares. The dot product ⟨g1, g2⟩ multiplies matching entries and adds them up (the dot product and the cosine); dividing by both norms leaves only the angle ψ (psi). The cosine is 1 for the same direction, 0 at a right angle and negative beyond it; the PCGrad paper calls a negative cosine a conflict.
m ← 0.98 m + 0.02 cos ψ
Single cosines are noisy and often negative: in the Recon paper, Shi and colleagues (2023) measured 38% to 43% of them below 0 in plain joint training. So the log keeps a moving average m: each step keeps 0.98 of the old value and adds 0.02 of the new one. Keeping more, such as 0.99, smooths more but reacts later; 0.9 to 0.99 is a common range, a rule of thumb. Act when m stays below 0 for thousands of steps, also a rule of thumb.
So a cosine whose running value stays negative across steps, not one batch, signals real conflict: one batch at −0.707 moves m only from 0.30 to 0.280.
Look at m, the moving average of the cosine.
# n1, n2 from the rho check
dot = (g1 @ g2).item()
cos = dot / (n1 * n2).item()
# first logged step
m = cos
# every later logged step
m = 0.98 * m + 0.02 * cos
Figure 5.1 Steps cross 0; the average sits 2 above it if healthy, 3 below if failing.
For lasting conflict, try PCGrad, GradVac, CAGrad or MGDA, and judge them by the task losses; in most of Recon's tests they change how often conflicts occur only a little.
5.1 Conflicting gradients in a ravine
A ravine is a narrow valley with steep walls and a nearly flat floor. Liu and colleagues (2021), who introduced CAGrad, show gradient descent stuck in one when one task's gradient dominates.
Figure 6.1 Steps bounce across the walls and 2 crawl to 3 the minimum.
The task steps have a cosine of about −0.3 and task 1's is 3 times longer, so their sum points across the valley. A smaller learning rate stops the bouncing but slows the crawl; conflict methods shrink the opposing part instead.
5.2 When a conflict raises a task's loss
Yu and colleagues (2020), who introduced PCGrad, argue that conflict hurts mainly when it comes with a large norm gap and high curvature, meaning the loss surface bends sharply so a step that looks small can change the loss a lot; they call the three together the tragic triad. The two-task condition shows the norm part.
In the running example ρ = 2.828, so the threshold is −1 over 2.828 = −0.354. The cosine −0.707 is below it, so the summed step d = (−4, 3) should raise seg’s loss: its dot product with seg’s gradient is −8 + 3 = −5.0 (illustrative).
A conflict raises task 2’s loss once the cosine drops below minus one over ρ, so the larger task 1’s lead in size, the milder the conflict that already hurts task 2.
⟨g2, d⟩ = ‖g2‖2 (1 + ρ cos ψ)
cos ψ is the cosine of Section 5, and d is the summed step. Writing d as g1 plus g2 and expanding gives task 2's own part ‖g2‖2 plus the cross term with task 1, which is ‖g1‖ times ‖g2‖ times cos ψ. Factoring out ‖g2‖2 leaves the norm ratio ρ from Section 4 times the cosine in the bracket.
cos ψ > −1ρ
Task 2 still improves while the bracket is positive, which holds exactly when the cosine stays above minus one over ρ. With equal norms (ρ = 1) only exactly opposite gradients fail; at a 10× gap, any cosine below −0.1 raises task 2's loss.
So the test joins scale and direction into one check of whether the summed step hurts the smaller task: for seg, ‖g2‖2 = 5 and ρ cos ψ = 2.828 × (−0.707) = −2, so the first line gives 5 × (1 − 2) = −5.
Figure 7. At 120° apart, 1 equal norms let task 2 fall; 2 a 10× gap makes it rise.
6 Symptoms and their causes
Symptom
Likely cause
Why
First fix
One task dominates from the start
Losses on different scales
The step follows the task with the longest gradient (Section 4)
Papers report Δm, the average relative change against single-task models; positive means worse. It can look fine while one task is sacrificed.
Figure 8. Under MGDA, 1 segmentation falls 20%, yet 2Δm is only 1.38%.
7.2 Combining gradients over head weights
PCGrad, GradVac and CAGrad combine gradients over the shared weights only; each head takes its own task's gradient. Head weights in the vectors pull every cosine toward 0. GradNorm, Uncertainty Weighting and the official MGDA code scale whole losses, which rightly scales their head gradients too.
Figure 9.1 Each task’s nonzero head entries add length but nothing to the dot product, since the other task has 0 there, so 2 the cosine shrinks to −0.12.
7.3 Not renormalizing GradNorm's weights
GradNorm's loss weights wi must sum to the number of tasks K; otherwise the total drifts like a changing learning rate (why they sum to the task count).
Look at w.mul_, which rescales the weights to sum to K after each update.
import torch
# weights after a GradNorm step
w = torch.tensor([1.3, 0.9])
K = len(w) # number of tasks
with torch.no_grad():
w.mul_(K / w.sum())
# w.sum() is 2.0 again
7.4 The cost of one gradient per task
Methods that combine task gradients take one backward pass per task each step. CAGrad's fast variant samples a subset S of tasks and spends |S| + 1 passes, one per sampled task plus the average; on the 50 robot tasks of Meta-World MT50, 8 sampled ran about 5 times faster than PCGrad (CAGrad in practice). A larger |S| costs more and nears the full method.
Figure 10.1 The full method takes 50 passes; 2 sampling 8 takes 9.
8 Separate datasets and small task batches
Tasks with no shared examples still share the trunk, so their gradients can still conflict (Figure 7); what changes is noise, since each gradient averages only its own examples.
Figure 11 uses this setting: a task’s gradient measured on 8 examples per step spreads twice as wide as on 32, since the square root of 32 over 8 is 2. So if seg trains on 8 images per step beside depth on 32, with the same per-example spread, seg’s measured gradient is twice as noisy.
A gradient measured on a small batch is noisy, and the noise falls only with the square root of the batch size.
SD(gi) ≈ si√Bi
gi is the measured gradient of Section 2, and its noise feeds straight into the conflict test of Section 5.2. SD is the standard deviation, the typical size of the random error in each entry of gi from one batch to the next, and si is that standard deviation for a single example. Averaging Bi independent examples divides it by √Bi, so going from 8 to 32 examples halves it.
So cosines and ratios measured on the small task wobble, and the running average m of Section 5 is what steadies them: seg’s gradient on 8 images is twice as noisy as depth’s on 32.
Figure 11.1 The spread is 2× wider than 2 at 4× the examples; 3 the average holds.
Few examples per step inflate a task's norm and pull its cosines toward 0, so compare tasks with similar counts. Noise also misleads CAGrad, whose step favors the task the average helps least. Three fixes help, in this order.
Pick the combining weights from a moving average of each task's gradient, not one batch, at one extra copy per task. MoCo (Fernando and colleagues, 2023), unrelated to the contrastive MoCo, does this to remove the bias noisy batches cause.
Lower CAGrad's radius c, which sets how far the step may turn from the average gradient toward the worst-off task. At 0 CAGrad is gradient descent; the convergence guarantee needs c below 1, and very large values approach MGDA. The paper used 0.4 on NYUv2.
9 Joint batches and round robin
The checks and methods need every task's gradient at the same weights in one step, which only a joint batch gives. In round robin, one task per step, a step can raise another task's loss unseen.
Figure 12. Cards are steps, cells are examples; the line is task 2's loss.
Look at sampler2, which draws the small dataset with replacement so both loaders run equally long.
1Toy data, 1000 examples for task 1 and 100 for task 2.
2The small set is drawn with replacement, so both run 16 steps.
3A joint batch each step, with task_grad from Section 2.
As a rule of thumb, cap the small dataset at about 3 passes per pass over the largest. If memory is short, keep each task's gradient and combine once all are in.
10 Choosing which losses train the shared trunk
A loss belongs in multitask training only if the trunk's features need it; otherwise train it later on its own head, trunk frozen. The frozen-head test decides in four steps.
Train with one main loss per task; measure each metric.
Freeze the trunk, fine-tune each head with all its losses, and measure again.
If the head reaches the metric you need, the extra losses stay with the head; if not, add that loss to multitask training.
Add losses one at a time.
Figure 13.1 The frozen head reaches the target; 2 for a trunk-level loss it falls short.
Each extra loss adds a gradient. MGDA steps opposite the minimum-norm blend of task gradients, which lowers every loss. Once the gradients point in enough different directions, the blend shrinks to zero and no step lowers them all.
Figure 14. Arrows at true length; a red ring marks a loss the average step raises.
11 Trunk capacity and task grouping
Balancing shares out what the trunk can represent; it adds nothing. If separate models still win on every task, try a wider trunk, soft parameter sharing (each task keeps its own layers and they exchange features), or separate trunks for tasks that keep conflicting. The intro page's table of which method fixes which failure links every method page.
12 References
Chen, Z., et al. (2018). GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. ICML. arXiv:1711.02257.
Kendall, A., et al. (2018). Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. CVPR. arXiv:1705.07115.
Sener, O., and Koltun, V. (2018). Multi-Task Learning as Multi-Objective Optimization. NeurIPS. arXiv:1810.04650.
Yu, T., et al. (2020). Gradient Surgery for Multi-Task Learning. NeurIPS. arXiv:2001.06782.
Liu, B., et al. (2021). Conflict-Averse Gradient Descent for Multi-task Learning. NeurIPS. arXiv:2110.14048.
Shi, G., et al. (2023). Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning. ICLR. arXiv:2302.11289.
Fernando, H., et al. (2023). Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approach. arXiv:2210.12624.