A logistic classifier says a post from the 20 Newsgroups forums is from the hockey forum with an illustrative chance of 0.7. A dropout keep mask, 1 to keep a unit and 0 to zero it, at drop chance 0.1 has chance 0.9 of a 1, so q counts keeps, not drops.
case
P(1)
P(0)
mean
variance
hockey label
0.7
0.3
0.7
0.21
keep mask
0.9
0.1
0.9
0.09
q is the chance of a 1.
P(x) =qx (1 − q)1−x
Capital X is the outcome, a random variable, and lowercase x is the value it takes, 0 or 1. The powers switch: x = 1 leaves q, x = 0 leaves 1 − q, so one formula covers both.
E[X] =q
The mean is the chance of a 1, the table’s 0.7 and 0.9.
Var(X) =q(1 − q)
The variance peaks at q = 0.5, the top of Figure 2, and vanishes at 0 or 1, when the outcome is certain.
Figure 2. The variance is largest when a 1 and a 0 are equally likely.
3 Softmax makes a categorical
The next token, a word or a piece of a word, is one of K choices, here the ten words of the decoding toy. A model gives each a raw score, a logitz, and softmax turns the scores into chances. Decoding and Transformer attention use it.
word
logit z
ez
chance, rounded
mat
2.0
7.389
0.431
floor
1.2
3.320
0.194
sofa
0.8
2.226
0.130
bed
0.5
1.649
0.096
the other six
0.1 to −2.6
2.564
0.150
total
17.148
1
Softmax turns any scores into positive chances that add to 1; only the gaps between scores matter.
P(k) =exp(zk)Σm exp(zm)
zk is choice k’s logit; exp(zk) raises e ≈ 2.718 to that logit, so it is always positive; the denominator adds the same term over every choice m. P(k) is choice k’s chance, the table’s last column: mat gets 0.431. Softmax, Temperature and Decoding writes the same chance with its own letters. Adding one number to every logit changes nothing. Figure 3 sets each logit beside its chance.
ΣkP(k) =1
The chances of all K choices add to 1, because the denominator is the sum of the numerators.
sigmoid(z) =11 + e−z
With two choices and logits z and 0, softmax gives the first one this chance: the sigmoid of logistic regression. So a Bernoulli is the K = 2 case of a categorical. At an illustrative z = 0.75 it gives 0.679, the bottom of Figure 3.
Figure 3. Mat leads floor by 0.8 in logit, so its chance is e0.8 ≈ 2.2 times floor’s.
Argmax picks the choice with the largest chance, mat here, which is greedy decoding; naive Bayes picks its class the same way. Temperature, a number that divides the logits first, flattens the chances when above 1 and sharpens them when below 1.
4 A uniform is flat over a range
An analog-to-digital converter (ADC), the chip that turns a microphone voltage into numbers, rounds each value to the nearest step. With an illustrative step Δ = 1 mV, one millivolt, its error lands anywhere from −0.5 to 0.5 mV with equal chance.
ADC error, Δ = 1 mV
value
height, so the area is 1
1 per mV
variance, standard deviation
0.0833 mV², 0.289 mV
The height is whatever makes the area 1.
p(x) =1xmax − xminfor xmin ≤ x ≤ xmax
xmin and xmax are the lowest and highest values x can take, so xmax − xmin is the width of the range and a wider range is a lower block (Figure 4). p(x) is the density, a height over values.
Var(X) =(xmax − xmin)²12
The variance grows with the width squared (Figure 4). For one ADC step the width is Δ, so the variance is Δ² over 12, the noise power the ADC page uses (Bennett 1948).
Figure 4. Twice as wide is half as high, and four times the variance.
5 The Gaussian, scaled and added
Diffusion adds Gaussian noise ε (epsilon) to an image step by step (Ho et al. 2020). Take one noise draw, ε = −1.0. The bell of Figure 5 gives its height and says how unusual it is.
A bell centred on μ with width σ, written N(μ, σ²).
r =x − μσ
x is one value, such as one noise draw. μ (mu) is the mean, σ (sigma) the standard deviation. N(0, 1) is the standard bell, mean 0 and variance 1. r, the standard score, counts standard deviations from the mean, the same scale for every bell.
p(x) =1√2πσ² exp(−r²2)
exp(·) raises e to the power. Minus r² makes the curve flat at the centre and fast to fall in the tails. The front factor makes the area 1, so a wider bell is lower; its π is the circle constant 3.14159. Figure 5 shades the area within one and two standard deviations. Φ(x) is the area left of x under N(0, 1). GMM writes this height at x as N(x | μ, σ²); the bar reads given the bell’s mean and variance.
Figure 5. 68.3% of N(0, 1) draws land within 1 of 0, and 95.4% within 2.
one illustrative noise draw
value
ε = −1.0 from N(0, 1)
r = −1, height 0.242, a height, not a chance
within one, two standard deviations
68.3%, 95.4%
area left of 1, Φ(1)
0.841
In an illustrative diffusion schedule, step t keeps a share 1 − βt of the signal’s variance and fills the rest with noise, with β = 0.05, 0.10 and 0.20 for steps 1 to 3. The kept shares multiply.
after step
kept share of the signal’s variance
noise variance
1
0.95
0.05
2
0.95 × 0.90 = 0.855
0.145
3
0.855 × 0.80 = 0.684
0.316
scaled by a number c
variance after
diffusion step 3 scales ε by √0.316 = 0.562
0.316
LayerNorm subtracts the mean of one token’s features and divides by their σ
Scaling a bell keeps it a bell; only its width and variance change.
Var(cX) =c² Var(X)
Stretching by c stretches the standard deviation by c and the variance by c². For a Gaussian, cX is still a bell, like the scaled left bell of Figure 6.
Figure 6. Two illustrative noise steps add up to one draw of variance 0.145.
independent sum
variance
two diffusion steps, step 1’s noise (variance 0.05) scaled by √0.9, plus fresh noise of variance 0.10
0.045 + 0.10 = 0.145, which is 1 − 0.95 × 0.90
a query-key score, 64 products of variance 1
64
a layer with 512 inputs and 512 outputs, inputs of variance 1, independent mean-0 weights of variance 1 over 512 (Glorot and Bengio 2010 use 2 over inputs plus outputs)
1; He et al. (2015) use 2 over 512 for ReLU, which zeroes negative inputs
Adding independent bells gives a bell; the variances add, not the standard deviations.
X1 + X2 ~N(μ1 + μ2, σ1² + σ2²)
X1 ~ N(μ1, σ1²) and X2 ~ N(μ2, σ2²) are independent (~ reads “is drawn from”). Means add; variances add because the covariance is 0. That is why two diffusion steps collapse into one draw, Figure 6 and the table’s 0.145.
Sums of many small independent effects tend toward a bell (the central limit theorem), so the mixed sources of independent component analysis (ICA) look more Gaussian than each source.
6 The Student-t has fat tails
To score how close two points sit on its 2-D map, t-SNE uses a curve that falls slowly, so far points still count and clusters can pull apart. That curve is the Student-t.
distance from the centre
Student-t
bell, N(0, 1)
0
0.318
0.399
3
0.032
0.0044
Figure 7. At distance 3, 1the Student-t is 0.032 and 2the bell 0.0044.
A bump like the bell whose tails fall like a power of the distance, not like exp.
p(x) =1π (1 + x²)
This is the Student-t whose degrees of freedom ν (nu) equal 1. ν sets how fat the tails are: raising it thins them, and as ν grows the curve becomes the bell N(0, 1). The π makes the area 1. At distance 3 the height is seven times the bell’s, Figure 7.
bij =11 + dij²
bij is t-SNE’s closeness of map points i and j at distance dij, the same curve without the π. t-SNE then divides it by the total over all pairs. At distance 3 it gives 0.1.
aj|i =exp(−dij²2)
aj|i, t-SNE’s closeness of point j to point i in the original high-dimensional space, is a bell-shaped closeness curve, a kernel, with width 1 here. It gives only 0.011 at distance 3, about a ninth of bij.
7 The multivariate Gaussian
Two side-by-side pixels at the centre of the Olivetti face photos that PCA uses rise and fall together. Read at the photos’ full 64 by 64 size and standardized over all 400 photos, shifted to mean 0 and scaled to variance 1, they have correlation 0.94, so their scatter of points, the cloud, tilts.
A bell over several numbers at once, written N(μ, Σ); the cloud tilts when they move together.
Σ =σ1²σ12σ12σ2²
A point x = (x1, x2) holds two numbers, such as one pixel pair. The bold Σ is the covariance matrix, not a sum. Its diagonal holds the variances σ1² and σ2², and its off-diagonal the covariance σ12 = Cov(X1, X2).
r² =(x − μ)TΣ−1 (x − μ)
μ is the cloud’s centre. r² is a squared distance in the cloud’s own units; T, the transpose, lays the column of numbers flat as a row. Σ−1 is the inverse of Σ: it undoes the cloud’s stretching, so a step along the long direction costs little. A step across the tilt costs more: (1, −1) is as long as (1, 1) but scores 33.3 against 1.03, Figure 8.
p(x) =1√(2π)n det Σ exp(−r²2)
n is the number of dimensions, and det Σ, the determinant, is how much Σ stretches area. With n = 1 and Σ = σ² this is the 1‑D bell of section 5. The cloud’s area grows with the square root of det Σ, and the front factor divides by that root, so a narrow cloud has a high peak. The round cloud, whose Σ is the identity matrix I with variance 1 on each axis and no tilt, peaks at 0.159; the face-pixel cloud peaks at 0.466. A diagonal Σ makes the density a product of 1‑D bells, so the coordinates are independent. GMM’s diagonal option uses that, and its spherical option one variance for every axis.
Figure 8. Squared distance r² for green (1, 1) and red (1, −1) in each cloud.
The table adds a second untilted cloud, diag(4, 0.25), with variances 4 and 0.25 on the diagonal and zeros off it.
cloud Σ
det
point
squared distance r²
10.940.941 face pixels
0.116
(1, 1), along the tilt
1.03
same
0.116
(1, −1), across it
33.3
I, round; diffusion’s ε ~ N(0, I)
1
(1, 1)
2
diag(4, 0.25)
1
(2, 0.5)
1 + 1 = 2
The face-pixel cloud is longest along (1, 1). Eigenvectors, taught in PCA, find such directions for any covariance matrix, and how much it stretches along each.
The Wiener filter guesses clean speech from what a microphone hears, using such a cloud. Speech S ~ N(0, σS²) and noise D ~ N(0, σD²) are independent. Both have mean 0, so each variance is that signal’s power, the speech and noise powers of Wiener Filtering. The microphone hears their sum Y, so speech and what is heard form a tilted cloud. Take σD² = 1.
Seeing Y shifts the guess about S and narrows it.
ξ =σS²σD²
ξ (xi) is the speech-to-noise power ratio; with σD² = 1 it is the speech variance itself.
Σ =σS²σS²σS²σS² + σD²
The covariance of (S, Y). The bottom right is Y’s variance: all of S’s power plus the noise’s. The off-diagonal σS² is Cov(S, Y): the noise is independent of speech, so the only part S and Y share is S itself.
E[S | Y] =σS²σS² + σD² Y
E[S | Y] is the average of S over all cases with that heard Y, the best guess of the speech. The fraction is the share of Y’s power that is speech, the same as ξ over (1 + ξ). It is the Wiener gain, the slope of the guess against Y, drawn dashed in Figure 9.
Var(S | Y) =σS² σD²σS² + σD²
The variance remaining after hearing Y, always below σS². At ξ = 3 it is 0.75, a standard deviation of 0.866, the give-or-take of Figure 9.
ξ
gain (slope)
variance remaining
guess at Y = 2
3
0.75
0.75
1.5
1
0.5
0.5
1.0
0.33
0.25
0.25
0.5
Figure 9. The slope of the dashed line is the Wiener gain.
8 Losses are minus log-likelihoods
The likelihood is the chance a model gives to the data it saw. Training minimizes its minus natural log, −ln. The ln of a number is the power of e ≈ 2.718 that gives it, so a chance below 1 has a negative log and a positive minus log.
model says
what happened
minus log
hockey with chance 0.7 (Bernoulli)
hockey
−ln 0.7 = 0.357
same
not hockey
−ln 0.3 = 1.204
mat with chance 0.431 (categorical)
next token is mat
−ln 0.431 = 0.84
prediction ŷ, bell with σ = 1 (Gaussian)
true value y, with y − ŷ = 1
0.5 + 0.919 = 1.419
Each minus log is a training loss.
−ln N(y | ŷ, 1) =(y − ŷ)²2+ 12 ln 2π
y is the true value, a training target, not a value of the heard Y of section 7; ŷ (y-hat) is the model’s prediction, used as the bell’s mean. The first term is half the squared error. The second, one half of ln 2π, is 0.919, a constant that does not change which prediction is best, so minimizing this is minimizing squared error (regression).
−ln P(y) =−y ln q− (1 − y) ln(1 − q)
y is again the true value, here a label: 1 for hockey and 0 otherwise. q is the model’s chance of hockey; the cross-entropy page uses another letter for it. Only one term survives for each label. This is binary cross-entropy, the BCE loss.
CE =−ln P(k)
For a categorical, k is here the true choice, one of the K, and the minus log of its chance is the cross-entropy loss CE, averaged over a batch of examples. Figure 10 plots the minus log: a chance near 0 costs without limit.
Figure 10. A confident miss costs far more than a hit; near 0 the loss has no ceiling.
Decoding draws the next token (sampling). In reinforcement learning the learner, called the agent, draws its action from its policy, often a softmax over actions (policy gradients). k-means++ draws its starting cluster centres. Subword sampling draws one way to split a word into tokens. A categorical draw picks u uniformly between 0 and 1 and takes the first word whose running total passes it.
draw
result
an illustrative u = 0.70, running totals 0.431, 0.625, 0.754
sofa, since 0.625 < 0.70 ≤ 0.754
diffusion step 3 noise, 0.562 times an illustrative ε = −1.0
−0.562
DDPG adds exploration noise to a robot arm’s torques, the twisting forces at its joints. The noise, Ornstein-Uhlenbeck (OU), is pulled back toward 0, and each step’s random kick is σ times a fresh ε
one kick per torque, each step
Any Gaussian draw is a standard one, scaled and shifted.
x =μ + σ ε
ε is a standard-normal draw, mean 0 and variance 1. Scaling by σ and shifting by μ gives a draw from N(μ, σ²), by the scaling rule of section 5. Figure 11 shows a categorical draw and this scaled draw.
Figure 11.u = 0.70 lands on sofa; ε = −1 scales to −0.562.
torch.normal takes the standard deviation, not the variance, so step 3 passes 0.562, not 0.316. The snippet builds the categorical, the face-pixel cloud and a scaled draw in torch.distributions.
2The decoding toy’s logits z as a categorical, which applies softmax itself. z.argmax() is mat, so logprob is about −0.84, the log of 0.431. log_prob works in logs, so a tiny chance does not round to 0 first.
3The face-pixel cloud of section 7 and one draw of a pixel pair from it.
4The equation above with μ = 0 and σ = 0.562 from the diffusion schedule.
10 References
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Sec. 2.1 to 2.3. PDF.
Goodfellow, I., Bengio, Y. and Courville, A. (2016). Deep Learning, Sec. 3.9, Common Probability Distributions. MIT Press.
Bennett, W. R. (1948). Spectra of quantized signals. Bell System Technical Journal, 27.
Samaria, F. and Harter, A. (1994). Parameterisation of a stochastic model for human face identification. IEEE Workshop on Applications of Computer Vision (the Olivetti faces, as packaged in scikit-learn).