A matrix turns most arrows it multiplies. A few directions only get longer or shorter. These are its eigenvectors, and the stretch of each is its eigenvalue. For face photos, the eigenvectors of the pixels’ covariance (Section 3) are the directions the pixels vary along most. PCA turns them into face-shaped directions, the eigenfaces. Collapse checks test that a model’s output vectors have not squeezed onto a few directions. They list the eigenvalues, the spectrum. LoRA fine-tunes a model with a low-rank update, one built from a few directions.
1 The few directions that matter
Figure 1. Face pixels 1, an eigenface 2, a spectrum 3 and a rank-1 table 4.
2 Directions a matrix only stretches
PCA’s eigenfaces and the axes of GMM’s ellipses are these directions. The example is R, the correlation table of two face pixels, rebuilt from the 24 eigenfaces that PCA stores for its 400 photos, 10.610.611: 1 on the diagonal, and 0.61, how strongly the pixels brighten together (1 would mean always). Vectors and Matrices, section 8 uses the same table.
input
R times it
what happened
(1, 1)
(1.61, 1.61)
stretched by 1.61
(1, −1)
(0.39, −0.39)
stretched by 0.39
(1, 0)
(1, 0.61)
turned by 31.4°, not an eigenvector
Figure 2. Two arrows keep their line and only change length; (1, 0) turns.
A direction the matrix only stretches is an eigenvector; the stretch is its eigenvalue.
Mv =λv
M is any matrix and v a direction, an arrow from the origin. Mv is the matrix’s output for it (Vectors and Matrices, section 5). λ (lambda) is the stretch: the output points along v, only λ times as long. In Figure 2 the matrix is R, and (1, 1) is an eigenvector with λ = 1.61, while (1, 0) is not, because its output turns.
Section 4 finds every such pair for a covariance at once.
3 Covariance from data
PCA and collapse checks build this matrix, the covariance, here written C.
Figure 3. The four faces around their mean, and the ellipse of their covariance.
The covariance averages, over faces, each centred face times itself as an outer product.
C =1N Σj(xj − x̄)(xj − x̄)T
xj holds face j’s two pixel values and x̄ (x bar) their mean over the N faces, so xj − x̄ is face j centred, the green arrow in Figure 3. Times its own transpose it makes a 2 × 2 table, the outer product of Vectors and Matrices, section 6.3. Σj adds the N tables and 1N averages them. The diagonal of C holds each pixel’s variance, its average squared distance from the mean; the off-diagonal says how the two move together.
Four faces give a rough correlation, 0.874. PCA’s 24 stored eigenfaces give 0.61, the table R of Section 2.
4 Every covariance splits into its directions
R is itself a covariance: the covariance of the two pixels after each is scaled to variance 1. The scaling moves the 51° tilt of Figure 1 to 45°, the direction (1, 1).
for R, direction
variance
reading
(1, 1) over √2
1.61
the larger λ, λ1, both pixels brighter
(1, −1) over √2
0.39
the smaller, λ2, one brighter, one darker
both together
2
1 + 1, the diagonal’s sum, so λ1 holds 1.612 = 80.5%
A covariance splits into its eigenvectors and eigenvalues: turn to the eigen-directions, stretch each, turn back.
C =VΛVT
V holds the eigenvectors v1 and v2 as columns; Λ (capital lambda) holds the eigenvalues λ1 ≥ λ2 on its diagonal, 0 elsewhere. Reading right to left, VT turns a point onto the eigen-directions, Λ stretches each by its λ, and V turns it back. For R, the columns of V are (1, 1) and (1, −1) over √2.
eTCe ≤λ1
e is any direction of length 1 and eTCe the variance along it, the table’s variance column for R. It peaks at λ1 when e is v1, so PCA’s first eigenface is the direction of most variance. A diagonal C has the axes as eigenvectors; equal eigenvalues make a round cloud, the diag and spherical shapes of GMM.
VTV =I
I is the identity, 1 on the diagonal and 0 elsewhere. So V is orthogonal: its columns have length 1 and meet at right angles, and it keeps every length, only turning or flipping. RoPE turns each query and key that attention compares with such a matrix, by an angle set by its position. A turn keeps lengths, so the dot product changes with the positions only through the difference of the two angles, which is set by the gap between them.
Dividing each point’s coordinate along each eigen-direction by the √λ of that direction gives variance 1 in every direction, which is whitening (PCA variants); ICA starts from whitened data.
stretches the first axis by 3.162 and the second by 0
U
turns the first axis toward (2, 1) over √5
result
the circle becomes a segment of half-length 3.162, the most it stretches any input, the spectral norm
Figure 4. The update turns, stretches one direction by 3.162 and flattens the other.
The SVD of the four centred faces of Section 3 gives their covariance’s eigenvalues.
four faces, centred
first
second
singular value s
0.2929
0.0703
s2 over 4
0.02145
0.00124
eigenvalue of their C
0.02145
0.00124
direction, a row of VT
(0.547, 0.837)
(−0.837, 0.547)
Any matrix is a turn, a stretch along the axes, and another turn.
M =USVT
VT turns the input so the directions M stretches most lie on the axes. S is diagonal and stretches axis i by the singular valuesi ≥ 0, largest first. U turns the result into place. U and V are orthogonal, and a zero si flattens its direction, as in panel 3 of Figure 4.
λi =si2N
Take X, the centred faces, one per row. Its V holds the eigenvectors of their covariance, the same V as in Section 4, and each eigenvalue is a squared singular value over N, as the table shows for the four faces. So PCA in practice gets the eigenfaces from one SVD, without building C.
25.39% of the variance at k = 1, 63.38% at 7, 82.4% at 24, 90% at 48
collapse, 200 illustrative points in 3-D
eigenvalues 1.499, 1.029 and 0, so rank 2, and the points lie on a plane
ratings, illustrative, a row per user and a column per item
the column (1, 2, 0.5) times the row (2, 1) gives 214210.5, rank 1
LoRA, rank 8 on 768 × 768
12,288 numbers instead of 589,824, 2.08%
Keep the k largest stretches and drop the rest.
M ≈Σi = 1ksiuiviT
ui and vi are column i of U and V, so uiviT is one outer product, a rank-1 block stored as a column, a row and its weight si. The sum from i = 1 to k adds the first k blocks; ≈ (nearly equal) leaves out the rest, which is the error. The first k give the best rank-k copy in squared error (Eckart and Young, 1936). Blocks may hold negative numbers that cancel. NMF (non-negative matrix factorization) keeps every block non-negative, so the blocks add up as parts.
kept share =Σi = 1ksi2Σisi2
The share of M’s squared size (the sum of its squared entries) that the first k blocks keep. For centred data each si2 is N times an eigenvalue, so this is the eigenvalue share, 25.39% at k = 1 for PCA’s faces (the blue line in Figure 5).
Figure 5. Face variance piles into a few directions; collapse zeroes an eigenvalue.
One photo is also a matrix, 32 × 32 brightness values. Minus its average brightness, its own SVD rebuilds it block by block; PCA’s reconstruction tool instead uses eigenfaces shared by all 400 photos.
4
Figure 6. A few column-times-row blocks rebuild a face; drag the bars.
7 Eigenvectors and SVD in NumPy
The block runs eigh on R and svd on the four centred faces. Its printed numbers match the tables in Section 3, Section 4 and Section 5.
import numpy as np
R = np.array([[1.0, 0.61],
[0.61, 1.0]]) # face-pixel table
lam, V = np.linalg.eigh(R) # smallest first
print(lam, V[:, -1].round(3)) # [0.39 1.61] [0.707 0.707]
faces = np.array([[0.654, 0.736], [0.620, 0.584],
[0.793, 0.774], [0.561, 0.463]])
X = faces - faces.mean(axis=0) # centre first
U, s, Vt = np.linalg.svd(X, full_matrices=False)
print((s**2 / 4).round(5)) # [0.02145 0.00124]
print(np.cov(faces.T, bias=True).round(5)) # C, over N
1The eigenvalues of R (Section 4); the top direction is the last column.
2The four faces’ eigenvalues from the SVD (Section 5), without building the covariance.
3The covariance itself (Section 3); bias=True divides by N.
trap
what goes wrong
fix
sign and order
an eigenvector can come back as its negative; eigh lists smallest first, svd largest first and returns VT
Strang, G. (2016). Introduction to Linear Algebra, 5th edition, Chapters 6 and 7. Wellesley-Cambridge Press.
Goodfellow, I., Bengio, Y. and Courville, A. (2016). Deep Learning, Sections 2.7 and 2.8. MIT Press.
Eckart, C. and Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1(3), 211 to 218.
Turk, M. and Pentland, A. (1991). Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1), 71 to 86.
Samaria, F. and Harter, A. (1994). Parameterisation of a stochastic model for human face identification. IEEE Workshop on Applications of Computer Vision.