On this page
Find which kind of error is largest before choosing a fix.
A model's error on new data is the sum of four parts, each with its own cause and fix, and two curves measured from your own runs show which is largest. The running example is a gesture model trained on sensor signals recorded from people.
Figure 1. Four parts of the error, each with its fix; find the largest before fixing.
The model class is every function the model form can represent. A true function outside it is never found, as a straight line cannot fit a curve (Regression); the fix is a more expressive model, such as a deeper MLP.
An expressive class can still see too little data. Overfitting is memorizing the training data, noise and all, so the model fails on anything new; its sign is the generalization gap:
The fixes are more data, regularization or augmentation.
Training can still miss the best weights: it stops in a local minimum or moves very slowly near a saddle point, a flat point that is not a minimum.
The fix is a better optimizer or schedule.
Irreducible error is the part no model, data or optimizer removes, because the noise lives in the data:
A second kind of sensor is one source of such new information.
On the handwritten-digit benchmark MNIST, error that looked irreducible was not (LeCun et al. 1998; Ciresan et al. 2012, trained on distorted digits; Wan et al. 2013; Northcutt et al. 2021).
Figure 2. MNIST test error by paper; the dashed line marks 15 wrong labels out of 10,000.
Later models closed most of the 1998 gap, so most of it was never irreducible.
Approximation and estimation error trade off: the bias-variance trade-off. Bias is the error of a model too simple for the data, the same on any sample; variance is the error of a flexible model that fits its sample's noise, so a new sample gives a new model.
Figure 3. Toy curves; bias falls, variance rises, and marker 1 is the lowest test error.
To find the largest part on a real model, measure error against model capacity (how much the model can represent) and against training-set size.
Figure 4. Each curve shape points to the largest error; the fix follows the arrow.
Figure 5. Toy curves; more data lowers test error and moves its minimum (the dot) right.
Both curves are drawn after training, never inside the loop: each point is one full training run per cross-validation fold, so the capacity curve trains 4 depths on 5 folds, 20 models.
validation_curve trains the decision tree clf on 500 noisy two-moons points at each max_depth and returns accuracy per depth and fold.
import sklearn.model_selection
from sklearn import datasets, tree
moons = datasets.make_moons
X, y = moons(500, noise=0.3)
dt = tree.DecisionTreeClassifier
clf = dt()
depths = [2, 4, 8, 16]
grid = dict(param_range=depths)
grid["param_name"] = "max_depth"
ms = sklearn.model_selection
vc = ms.validation_curve
tr, te = vc(clf, X, y, **grid)
tr_err = 1 - tr.mean(axis=1)
te_err = 1 - te.mean(axis=1)For the data curve, learning_curve retrains clf at its default five sizes, 10 to 100 percent of each training fold.
lc = ms.learning_curve
n, _, te_d = lc(clf, X, y)
# test error at each size n
err_d = 1 - te_d.mean(axis=1)The irreducible-error check passes only when the last capacity step, the last data step and the train-to-test gap all stay under tol, so it tests that the curves stopped falling, not that they are small; a smaller tol demands flatter curves.
tol = 0.005 # half a percent
# each curve's last drop
d_cap = -np.diff(tr_err)[-1]
d_data = -np.diff(err_d)[-1]
gap = te_err[-1] - tr_err[-1]
worst = max(d_cap, d_data, gap)
at_floor = worst < tolA wearable model is trained on recordings from a group of people and used by others. A random split puts the same person in training and test, so the test score looks better than it will on a new person. Hold out whole people instead, one per fold (LeaveOneGroupOut); the worst per-person score shows how a new person may fare. The code reads the 500 rows as 10 people of 50.
It costs one full training run per held-out person, once, when choosing or checking a model.
# person id per row: 10 x 50
ids = np.arange(10)
groups = np.repeat(ids, 50)
logo = ms.LeaveOneGroupOut()
cv = logo.split(X, y, groups)
cvs = ms.cross_val_score
acc = cvs(clf, X, y, cv=cv)
print(acc.min(), acc.mean())| Test | What it measures |
|---|---|
| Offline | cross-validation across people: accuracy on a person never seen |
| Online streaming | latency, throughput and false alarms per hour, with data arriving live |
| Stress | accuracy and recovery under added noise or a shifted sensor placement |
A new person who still fails can get a small trainable add-on that adapts the model without retraining it (Adapters). Per-person normalization needs no labels (z-score per person, preprocessing); the rest is under Cleaning sensor data.
On a real project the curves and the per-person test combine into one learning curve whose folds hold out whole people, with preprocessing refit inside each training fold. It reuses clf, X, y, lc and groups from Section 5 and Section 6.
import sklearn.pipeline as pl
import sklearn.preprocessing as pp
# refit on each training fold
sc = pp.StandardScaler()
pipe = pl.make_pipeline(sc, clf)
gkf = ms.GroupKFold(5)
cv = gkf.split(X, y, groups)
res = lc(pipe, X, y, cv=cv)
n, tr, te = res
tr_err = 1 - tr.mean(axis=1)
te_err = 1 - te.mean(axis=1)It costs one training run per size and fold, five default sizes (10% to 100%) times five folds, 25 runs (scikit-learn learning_curve documentation), once, never inside training.
groups works only with a group splitter such as GroupKFold; a plain 5-fold cv ignores it and mixes people again.learning_curve does not shuffle by default, so small sizes take the first rows; rows sorted by person give them only a few people.Overfitting, training error sinking while test error climbs, is reduced by constraining the model or adding data: Regularization and Data Augmentation. How error falls as data and models grow is on Scaling Laws.