On this page
Methods that stop a model fitting the noise in its training data instead of the pattern.
A model overfits when training error sinks while test error climbs (Understanding Model Errors). Two families of methods reduce it, constraining the model and adding training data; a third part cleans the data so the model learns the signal, not sensor faults. All act only in training or on the data and are judged on held-out data.
Dropout zeroes each activation with probability p on every training forward pass (one run of an input through the network), so each pass trains a thinned sub-network and no single neuron can carry the prediction; at test time the full network runs. The mask is a Bernoulli draw, a weighted coin flip per neuron that lands on "keep" with probability 1 − p.
Each example gets a fresh mask; there is no state, and the cost is one random draw and one multiply per activation (Srivastava et al. 2014). A larger p regularizes more and underfits when too high; p = 0 turns dropout off, and at p = 1 nothing passes. PyTorch's default is 0.5.
Dropout for one layer of 12 activations; a_tilde is ã, and 0.2 is a common rate.
import torch
p = 0.2 # drop chance
a = torch.randn(12)
u = torch.rand_like(a)
m = (u > p).float() # mask
a_tilde = m * a / (1 - p) # ãAt inference, nothing is dropped: the full network runs, and the training-time rescale already matches its scale. torch.nn.Dropout masks only in training mode; model.train() and model.eval() switch every dropout layer at once.
drop = torch.nn.Dropout(p)
drop.train()
a_train = drop(a) # masked
drop.eval()
a_test = drop(a) # equals aDropout acts like noise injection in the hidden layers: the noise forces features that survive small disturbances. Rates are often small; the original Transformer used 0.1. Stochastic depth skips whole residual blocks (Huang et al. 2016), and DropConnect zeroes single weights instead of units (Wan et al. 2013).
Figure 1. Higher rates drop more units and scale the rest up to keep the dashed average.
A penalty on the weights, added to the loss, makes large weights cost something, so training trades a little fit for smaller weights, and with L1 many exactly zero. It acts at every step on every weight, keeps no state and costs one sum over the weights.
L2 produces shrinkage: the weights get smaller but stay non-zero. With Adam, use AdamW, which applies it as weight decay in the optimizer step.
L1 regularization uses absolute values instead and produces sparsity.
Read as a constraint, the weights stay inside a ball (L2) or a diamond (L1) around the origin, and the solution is where the loss contours (rings of equal loss) first touch it.
Figure 2. L2 keeps both weights nonzero; L1 hits the contours at a corner, one weight zero.
Regularization cuts the estimation error, the error from too little data for the model's flexibility (Understanding Model Errors). More varied training data also cuts it, leaving less room to memorize.
A gesture recorded with the wristband twisted is still the same gesture, so a twisted copy is a new example with a true label; each augmentation in the table mimics one real source of variation. The same label-keeping edits make the positive pairs of Contrastive Learning.
Augmentations are drawn fresh each time a training window is loaded, so every epoch sees a new copy; they touch training data only, keep no state, and cost data-loader time, not model time.
| augmentation | what it mimics |
|---|---|
| channel dropout | lost contact or a misaligned sensor |
| channel shuffling | the band shifting along the arm |
| spatial mixing | the displacement blur of a shifted band |
| rotation | the band worn twisted or at another angle |
| random scaling | strap tightness or skin-electrode impedance (how much the skin contact holds back the signal) |
| temporal jitter | the delay that altered placement introduces |
| cross-modal masking | a missing sensor type |
Twisting the band keeps x and rotates y and z, so the augmentation rotates the readings.
SciPy rotates a window by a fresh small random roll, pitch and yaw; acc_rotated keeps the shape [time, 3].
import numpy as np
import scipy.spatial.transform
rng = np.random.default_rng()
shape = (200, 3) # [t, 3]
acc = rng.normal(size=shape)
# roll, pitch, yaw in degrees
deg = rng.uniform(-15, 15, 3)
rad = np.radians(deg)
st = scipy.spatial.transform
euler = st.Rotation.from_euler
rot = euler("xyz", rad)
acc_rotated = rot.apply(acc)Channel dropout and shuffling, each into a new array.
import numpy as np
rng = np.random.default_rng()
x = rng.normal(size=(8, 500))
n_ch, p = len(x), 0.1
# channel dropout: zero rows
keep = rng.random(n_ch) > p
x_drop = x * keep[:, None]
# shuffle: swap rows i, i + 1
i = rng.integers(n_ch - 1)
rows = [i, i + 1]
x_shuf = x.copy()
x_shuf[rows] = x[rows[::-1]]Random scaling and jitter; wider ranges regularize more, until they make signals no real wearer would record.
# strap tightness per channel
u = rng.random((n_ch, 1))
gain = 0.8 + 0.4 * u
x_scaled = x * gain
# jitter: shift ≤ 20 samples
k = rng.integers(-20, 21)
x_jit = np.roll(x, k, axis=1)If 95 of 100 windows are rest, always guessing rest scores 95 percent and catches nothing: class imbalance makes the model overfit to easy, frequent samples. You can oversample minority classes, undersample majority classes, or augment minority classes more. SMOTE (Synthetic Minority Over-sampling Technique, Chawla et al. 2002) oversamples without duplicating: it finds a nearby minority sample with k-NN (k-Nearest Neighbors) and interpolates a new one between them.
Class-weighted loss weights the minority class higher; balanced batch sampling gives each batch (a group of examples processed together) roughly equal classes. In the code 95 labels are common and 5 rare, and w is the inverse class frequency, used by the loss or the sampler, not both.
from torch import nn, tensor
import torch.utils.data as tud
labels = [0] * 95 + [1] * 5
y_train = tensor(labels)
counts = y_train.bincount()
w = counts.sum() / counts
# option 1: weighted loss
loss_fn = nn.CrossEntropyLoss(w)
# option 2: balanced batches
WRS = tud.WeightedRandomSampler
n = len(y_train)
sampler = WRS(w[y_train], n)The weights are computed once from the training labels; a weighted loss applies them every step, a weighted sampler every epoch (PyTorch's CrossEntropyLoss and WeightedRandomSampler). SMOTE runs once, on the training split only.
Figure 5. A larger w catches more rare windows and flags more common ones.
Focal loss (Lin et al. 2017) down-weights the easy examples the model already classifies confidently, so the hard ones dominate the loss:
Wearable signals arrive noisy, drifting and sometimes missing, and a model fed raw signals can learn the sensor faults.
A powerline hum on every channel and a slow zero-level drift are removed by a little signal processing.
Filtering runs on each recording before the model, the same for training, test and live data; the z-score keeps only a mean and a spread per channel of one person's recording.
| step | what it removes |
|---|---|
| band-pass filter | low-frequency drift and high-frequency noise, keeping one band |
| notch filter | one narrow band, the powerline hum, leaving the frequencies around it |
| rectify and envelope | the sign and the fast oscillation: absolute value, then a low-pass, traces how strong the signal is |
| z-score per channel, per person or session | amplitude differences across people and sessions: subtract that recording's average, divide by its spread |
filtfilt runs each filter forward and backward, so nothing shifts in time; the band-pass is 4th order, its edges fractions of the Nyquist rate nyq (half the 1000 Hz sampling rate).
import numpy as np
from scipy.signal import butter
from scipy.signal import filtfilt
from scipy.signal import iirnotch
fs, hum = 1000, 60 # Hz
x = np.random.randn(8, 2000)
nyq = fs / 2
wn = (20 / nyq, 450 / nyq)
b, a = butter(4, wn, "bandpass")
x_band = filtfilt(b, a, x)
b, a = iirnotch(hum, 30, fs)
clean = filtfilt(b, a, x_band)The envelope and the per-person z-score; z keeps the shape [channels, time].
# rectify, low-pass at 5 Hz
b, a = butter(4, 5, fs=fs)
rect = np.abs(clean)
env = filtfilt(b, a, rect)
# z-score per channel
mu = env.mean(axis=1)[:, None]
sd = env.std(axis=1)[:, None]
z = (env - mu) / (sd + 1e-8)Features are then hand-made in time and frequency, or learned by the model.
Bluetooth packet loss or sweat can leave missing channels. A channel with no spread, or at the top of the range (5 here) more than half the time, gets the mean of the good channels, and the mask bad goes to the model, so it learns to ignore the missing channels.
import numpy as np
x = np.random.randn(8, 500)
x[3] = 0.0 # a flat channel
flat = x.std(axis=1) < 1e-6
pegged = np.abs(x) >= 5.0
share = pegged.mean(axis=1)
bad = flat | (share > 0.5)
fill = x[~bad].mean(axis=0)
x_fill = x.copy()
x_fill[bad] = fillThe check runs on every window, in training and live, with no state; the mask travels with the window.
Combining modalities lets one sensor cover for another. An encoder compresses a signal into features; a decoder predicts from them.
A shared encoder needs the sensors aligned in time and scale; separate encoders do not. Very late fusion, one model per sensor with only the outputs joined, does not solve missing signals.
With cross-modal attention (Cross-Attention), one sensor's features query the other's, so the model learns how much to trust each and down-weights a missing or noisy one. A gives the queries, B the keys and values; fused has A's shape, w says how much each A step used each B step, and off marks B steps to ignore, in the default [time, batch, features] layout.
import torch
from torch import nn
# [time, batch, features]
A = torch.randn(50, 2, 32)
B = torch.randn(20, 2, 32)
off = torch.zeros(2, 20).bool()
off[:, 10:] = True # B drops
MHA = nn.MultiheadAttention
attn = MHA(32, num_heads=4)
# 4th argument: the B mask
fused, w = attn(A, B, B, off)If every B step in a row is masked, that row can come out as NaN; one fix is an always-valid token in B.
Foundation-style models such as SensorLM (Zhang et al. 2025) pretrain one encoder on a large cohort (Self-Supervised Learning); a small labeled calibration set from a new person then fine-tunes lightweight per-person heads.
Augmentation lives in the data pipeline, dropout in the model, weight decay in the optimizer step and class weights in the loss. The first snippet builds three once: a classifier net with dropout, AdamW, and a loss weighting a rare third class 5 times higher.
import torch
from torch import nn
net = nn.Sequential()
net.append(nn.Linear(8, 64))
net.append(nn.ReLU())
net.append(nn.Dropout(p=0.5))
net.append(nn.Linear(64, 3))
# AdamW default decay: 0.01
ps = net.parameters()
opt = torch.optim.AdamW(ps)
w = torch.tensor([1., 1., 5.])
loss_fn = nn.CrossEntropyLoss(w)The second runs one step on 32 rows; jitter stands in for augmentation (really in the dataset's __getitem__), and net.train() and net.eval() switch dropout on and off.
x = torch.randn(32, 8)
noise = torch.randn_like(x)
x = x + 0.05 * noise # jitter
y = torch.randint(3, (32,))
net.train() # dropout on
loss = loss_fn(net(x), y)
opt.zero_grad()
loss.backward()
opt.step() # also decays
net.eval() # dropout offNone costs much: dropout is one mask per forward pass and an identity in eval mode, weight decay one multiply per weight per step, class weights almost nothing, and augmentation CPU time in the loader every epoch. The dropout paper kept hidden units with probability 0.5 and inputs with 0.8. The focal loss paper used a gamma of 2 and an alpha of 0.25, where α weights the rare positive class against 1 − α, so raising it counts rare examples more; these are the defaults of torchvision's sigmoid_focal_loss. Dropout is torch.nn.Dropout.
weight_decay adds L2 to the gradient in SGD and Adam but shrinks the weights directly in AdamW.| symptom | likely cause | fix |
|---|---|---|
| Test predictions change between runs on the same input | dropout still on at test time | model.eval() before evaluating |
| The augmented model gets worse | an augmentation changes the label (a rotation that turns one gesture into another) | keep only transforms that leave the label true |
| The rare class is now over-predicted | class weights and a balanced sampler used together | use one of the two |
| The test score looks too good with SMOTE | synthetic samples made before the train and test split land near test points | split first, oversample the training part only |
Training must still lower the loss; the size and direction of each step is the optimizer's job, on Optimizers from SGD to Adam.