On this page
How test loss falls as data, model size and compute grow.
Once a model trains well with the controls in Optimizers in Practice: AdamW, SAM, Schedules and Clipping, how much data, model size and compute should it get? Scaling laws say test loss, the error on unseen data, falls as a power of the data, the parameters (the numbers inside the model, its size) or the compute. A fit runs once, after finished runs at several sizes, and costs only those runs; its compute-optimal split is chosen before the large run.
A power law, found empirically across many domains, raises the resource you grow to a power.
Compute is counted in FLOPs, multiplies or adds on floating-point numbers, about six per parameter per token for the forward and backward pass. The total a run may use is its compute budget.
An illustrative compute law; its exponent is near the 0.05 Kaplan et al. (2020) measured, and real coefficients come from your own runs.
Figure 1. Share of the gap kept on this page's toy laws, for each multiple of a resource.
Do not compare the exponents: the resources differ in units, and more compute buys both data and parameters.
Figure 2. The illustrative compute law drawn on log-log axes.
Figure 3. Every 10× of compute lowers the loss by less than 0.03.
If more data still lowers the test loss, the irreducible loss, the irreducible error no run removes, is not reached. Only the gap above it follows the power law, so only the right guess makes the gap straight on log-log axes.
Figure 4. The loss gap on log axes, for a guess too low and for the right guess.
The runs are made with irreducible loss 1.5 and true α = 0.50, which a good fit recovers. straightness scores a guess by how close log D and the log gap are to a line, 1 for a perfect one.
import numpy as np
# five runs: sizes, losses
D = np.logspace(3, 7, 5)
L = 1.5 + 2 / np.sqrt(D)
log_D = np.log(D)
def straightness(floor):
g = np.log(L - floor)
c = np.corrcoef(log_D, g)
return abs(c[0, 1])The sweep tries 50 guesses from 1.3 to just under the lowest loss, keeping every gap positive.
# keep every gap above 0
top = 0.9999 * L.min()
floors = np.linspace(1.3, top)
s = []
for f in floors:
s.append(straightness(f))
floor = floors[np.argmax(s)]
g = np.log(L - floor)
fit = np.polyfit(log_D, g, 1)
k, log_A = fit
alpha = -k # slope is −αFigure 5. The guess where straightness peaks is the irreducible loss.
A high irreducible loss means noisy data, so no model gets much lower; a large exponent such as 0.5 means more data still lowers the loss quickly.
The same law works over model size and compute; under a hard compute cap, as on a small device, it gives the best reachable loss.
At one compute budget a bigger model leaves fewer training tokens, so which split of the budget gives the lowest loss?
Models used to grow much faster than their data. Hoffmann et al. (2022) trained Chinchilla and showed that at a fixed budget, model size and training tokens (word-pieces of text, the unit the model reads) should scale together, about 20 tokens per parameter: 70B (billion) parameters on 1.4T (trillion) tokens.
The split in NumPy at 1023 FLOPs with the Chinchilla paper's constants: E is the irreducible loss L∞, A and B the constants, α and β the exponents.
import numpy as np
# the Chinchilla paper's fit
E, A, B = 1.69, 406.4, 410.7
alpha, beta = 0.34, 0.28
C = 1e23 # FLOPs
N = np.logspace(8, 12, 400)
D = C / (6 * N) # C = 6·N·D
L_N = A / N**alpha
L_D = B / D**beta
L = E + L_N + L_D
best = np.argmin(L)
ratio = D[best] / N[best]Too param-heavy, the model has more parameters than its tokens can train; too data-heavy, it is too small to use its tokens.
Figure 7. Loss against the split at one compute budget.
Figure 8. The curve of Figure 7; the readout gives your split's loss above the optimum.
The best ratio is very sensitive to how the fitted constants are rounded.
The law is fit once, to the final losses of finished runs; the Chinchilla paper fit all its constants together, minimizing a Huber loss on the log losses with L-BFGS from a grid of starts. The snippet does this in SciPy on the five runs; bounds keep the irreducible loss and A positive (so SciPy uses L-BFGS-B), and fit.x holds the irreducible loss, A and α.
import scipy.optimize as so
from scipy.special import huber
def cost(p): # floor L∞, A, α
floor, A, a = p
pred = floor + A / D**a
gap = np.log(pred / L)
h = huber(1e-3, gap) # δ
return h.sum()
x0 = [1.0, 1.0, 0.3]
pos = (0, None)
B = [pos, pos, (0.01, 2)]
opt = so.minimize
fit = opt(cost, x0, bounds=B)The fit takes seconds; each point is a full training run, and the Chinchilla paper trained over 400 models. Its Huber δ was 10−3 (a larger δ is closer to squared error, a smaller one ignores large misses more), robust to outliers and good at predicting held-out runs, with C ≈ 6·N·D (Chinchilla paper). It released no fitting code; the 2024 replication publishes data and code (replication paper).
fit.success and try several starts; the replication found the original intervals far too narrow, traced to an optimizer that stopped early.| symptom | likely cause | fix or reading |
|---|---|---|
| The fitted line bends at the right end | irreducible loss guessed too low | refit the irreducible loss (Section 2) |
| A fit on two nearby sizes | any slope fits two points | fit across sizes spread over several powers of ten |
| Not enough fresh data | repeating the same data | up to 4 epochs of repeats changed loss negligibly; beyond that, returns fade (Muennighoff et al. 2023) |
| A capability appears suddenly | loss is smooth, but capabilities can jump (emergence) | use the law to plan data, model size and compute, not to predict skills |