How models compare vectors and how matrices move them.
On this page
Models work on lists of numbers. Three tools carry almost every later page: length (clipping), the dot product and angle (attention, comparing task gradients) and matrices, which move a whole space at once (linear layers, turning a word’s vector by its position, small fine-tuning updates).
A vector is an ordered list of numbers, drawn as an arrow from the origin. Attention, convolutions, multitask learning and Multilayer Perceptrons (MLP) all store data this way. In shapes such as (B, T, d), each size names an axis: B counts examples, T positions in a sequence and d numbers per vector. A token is a word piece; its embedding is its learned vector.
A gradient says how the loss changes as each weight moves. The loss is the model's error score, and a weight is one learned number. Clipping in recurrent neural networks (RNNs), which read a sequence step by step, caps a gradient's length. GradNorm balances the gradient lengths of tasks trained together, and k-means clustering and k-nearest-neighbor (k-NN) search measure distances as lengths. PyTorch's clip_grad_norm_ rescales a gradient so its length is at most max_norm (Optimizers in Practice).
| clipping one illustrative gradient g = (3, 4) | result |
|---|---|
| length of g | √9 + 16 = 5 |
max_norm 1 | scaled by 1/5 to (0.6, 0.8), length 1, same direction |
max_norm 10 | unchanged, 5 is under the cap |
| a real model | every weight's gradient joins one long vector, capped as a whole |
Each token has a query, what it looks for, a key, what it offers, and a value, what it passes on. Attention scores keys against a query with dot products (Attention), and so does a convolution, which slides a small pattern of weights across an image. CLIP, a model that matches images to captions, and contrastive learning, training that pulls matching pairs together, score with the cosine, the dot product of the two vectors scaled to length 1 (below). The query and keys in this section are illustrative.
| query q = (2, 2) against key | entry by entry | score |
|---|---|---|
| cat (2, 2) | 4 + 4 | 8 |
| the (1, 0) | 2 + 0 | 2 |
| sat (−1, −0.5) | −2 − 1 | −3 |
Attention turns the raw scores 8, 2 and −3 into attention weights with the softmax, which makes every score positive and divides by their total, so the weights add to 1.
| key | score | key length | cosine |
|---|---|---|---|
| cat (2, 2) | 8 | 2.828 | 1 |
| the (1, 0) | 2 | 1 | 0.707 |
| sat (−1, −0.5) | −3 | 1.118 | −0.949 |
| the, doubled to (2, 0) | 4 | 2 | 0.707 |
The query’s length is 2.828. Doubling a key doubles its score and leaves its cosine alone.
CLIP scales every embedding to length 1 first, so one matrix product (Section 6) gives all its cosines at once. Attention skips that, so a longer key earns a bigger score.
Figure 2. Stretching the key by its square grip moves bar 1, not bar 2.
A task gradient is the gradient of one task’s loss with respect to the weights that several tasks share (Introduction to Multitask Learning). PCGrad removes the part of one task gradient that fights another, GradVac nudges the cosine between two task gradients toward a target, and multitask learning reads the fight from that cosine. PCA, which finds the main directions of a dataset, scores each image, such as a photo of a face, along one direction.
Every linear layer is this product: the hidden (inner) layer of an MLP, the query, key and value maps of Attention, and the weight row of logistic and linear regression. A matrix is a table of numbers, read as a stack of rows.
| an MLP's hidden layer on the point x = (−2, 2) | row of W times x | plus bias | output |
|---|---|---|---|
| row 1 (−1, 1) | (−1)(−2) + (1)(2) = 4 | −2 | 2 |
| row 2 (1, −1) | (1)(−2) + (−1)(2) = −4 | −2 | −6 |
The weights, the bias (−2, −2) and the point are those of the MLP walkthrough.
Score tables in Attention and Transformer Cost and Efficiency, padding masks, cross-attention, where one sequence attends to another, and pruning masks, which zero weights a model does not need, follow these rules.
Take six tokens at the original Transformer’s width 512: X has shape (6, 512). WQ, shape (512, 64), turns each token into a 64-number query, so dk = 64 is the query width. K stacks the keys the same way, S is the score table and h counts attention heads (parallel copies of attention).
In a batch, the axes before the last two ride along. An axis of size 1 stretches to match the other side, which is called broadcasting. @ is the matrix product sign in NumPy and PyTorch.
| operation | shapes | result |
|---|---|---|
| scores per head | (1, 8, 16, 64) @ (1, 8, 64, 16) | (1, 8, 16, 16) |
| padding mask (hides filler tokens) on the scores | (2, 1, 1, 5) with (2, 8, 5, 5) | (2, 8, 5, 5), the mask reaches every head and query |
| each point’s 2 cluster scores in a Gaussian mixture, a clustering model (Section 7), divided by their total | (272, 1) with (272, 2) | (272, 2) |
The sizes come from the code that turns each query and key by its position and the padding-mask code in Positional Encoding and Masking, and from the 272 points of the tool in Gaussian Mixture Models.
@ multiplies the last two axes and treats the axes before them, shown as …, as a batch. These are the scores per head from the table: 16 queries against 16 keys. With h attention heads, queries (B, h, T, dk) times transposed keys give one (T, T) score table per example and head.In attention, A holds the softmax weights, one row per query, and V holds the values, one row per token. A LoRA update is a small change added to a frozen (kept fixed) weight, built as a column times a row (Adapters).
| reading | example | result |
|---|---|---|
| a row of AV blends the rows of V | Attention's weights 0.78, 0.17, 0.05 on illustrative value rows (1, 0), (0, 1), (1, 1) | (0.83, 0.22) |
| two layers in a row are one layer | an illustrative rank-1 LoRA update (rank 1: every output lies on one line, Section 8) on x = (1, 1): the row (1, 1), then the column (2, 1) | the row gives 2, the column turns it into (4, 2); their product 2211 gives (4, 2) at once |
⊙ multiplies matching entries. The Adam optimizer, a rule for updating weights, keeps two running averages: one of the gradient, and v, one of its square (Adaptive Optimizers).
| operation | example | result |
|---|---|---|
| transpose KT | rows and columns of K swap | (6, 64) becomes (64, 6) |
| average gradient divided by √v, entry by entry | Adam, illustrative average gradient (1, 0.01) and v = (1, 0.0001) | (1, 1), gradients 100 times apart take the same step |
| column times row | error e = (−0.1604, 0) times input x = (−2, 2), computed from the MLP walkthrough’s network | 0.321−0.32100, the weight gradient |
A Gaussian Mixture Model (GMM) fits data with Gaussian bumps, bell-shaped clouds of points, called components. Its density, how likely a point is, divides by the square root of the determinant of each component’s covariance, its spread. When one component shrinks onto a point, that determinant nears 0. The likelihood, how well the model explains the data, then runs to infinity. LayerNorm rescales each feature (one entry of a vector) by a learned γ (The Transformer Block). RoPE turns a query or key by an angle set by its position (Positional Encoding and Masking).
| matrix | entries, row by row | det | the plane |
|---|---|---|---|
| illustrative γ = (2, 1) as a diagonal matrix | 2, 0, 0, 1 | 2 | areas double |
| RoPE turn by 40° | 0.766, −0.643, 0.643, 0.766 | 1 | areas, lengths and angles kept |
| rank-1 LoRA update | 2, 2, 1, 1 | 0 | lands on one line |
| the MLP walkthrough’s W from Section 5 | −1, 1, 1, −1 | 0 | both outputs read only the second input minus the first |
The LoRA update is illustrative too; 40° is one position’s turn in the RoPE figure of Positional Encoding and Masking. A diagonal matrix scales each axis on its own, as LayerNorm’s γ and a diagonal covariance in GMMs do.
np.linalg.det computes it, and the GMM density of this section needs it for its covariance, one row and column per feature.Figure 5. The shaded area is the determinant; lined-up columns give 0.
Linear regression’s direct formula for the best weights contains an inverse, RoPE needs a turn that can be undone (Positional Encoding and Masking), and LoRA (Adapters) and collapse checks, which test whether a model’s outputs have squeezed onto a few directions, count rank. Correlation says how strongly two pixels brighten together, 1 meaning always (Probability and Bayes Rule). The correlation 0.61 for two face pixels below is illustrative.
| matrix | det | inverse |
|---|---|---|
| two face pixels, illustrative correlation 0.61: 10.610.611 | 0.628 | 1.593−0.971−0.9711.593; times the matrix it gives the identity I (1s on the diagonal from top left to bottom right, 0s elsewhere) |
| rank-1 LoRA update 2211 | 0 | none |
| RoPE turn R by 40° | 1 | RT, the turn back |
At 40° per position, RoPE turns a query at position 3 by 120° and a key at position 1 by 40°; move both one place later and the 80° gap between them, and so their score, stays the same.
The rank is the number of independent directions left after the move; a rank-r matrix is a sum of r outer products (Section 6.3). The illustrative LoRA update is one outer product, the column (2, 1) times the row (1, 1), so its rank is 1. A rank-r LoRA update is a tall thin matrix with r columns times a short wide one with r rows. Embeddings that use few directions have low rank, the collapse the checks look for.
The code on the MLP page, Attention and contrastive learning use these calls and shapes.
| object | shape | note |
|---|---|---|
| one vector | (d,) | a 1-D array |
| a batch of vectors | (B, d) | one row per example |
| a batch of sequences | (B, T, d) | the input to attention |
| a linear layer’s weight W | (out, in) | used as the input times W transposed, plus the bias |
| attention scores | (B, h, T, T) | every query against every key |
PyTorch builds that weight with nn.Linear, given the number of inputs and the number of outputs. Attention scores multiply the queries by the keys after transpose swaps the last two axes of the keys.
Clipping and attention scores in NumPy. Clipping runs once per training step (one weight update), after the gradients are computed; scores run every time the model runs.
import numpy as np
g = np.array([3.0, 4.0]) # an illustrative gradient
max_norm = 1.0
length = np.linalg.norm(g) # 5.0
clipped = g * min(1.0, max_norm / length) # [0.6 0.8]
q = np.array([2.0, 2.0]) # query
keys = np.array([[2.0, 2.0], # cat
[1.0, 0.0], # the
[-1.0, -0.5]]) # sat
scores = keys @ q # [ 8. 2. -3.]
key_len = np.linalg.norm(keys, axis=1)
cosines = scores / (key_len * np.linalg.norm(q))
# [ 1. 0.707 -0.949]Layer and mask shapes in PyTorch. The layer runs every time the model runs; the mask is built once per batch.
import torch
import torch.nn as nn
x = torch.randn(6, 512) # 6 tokens, width 512
w_q = nn.Linear(512, 64, bias=False)
assert w_q.weight.shape == (64, 512) # (out, in)
assert w_q(x).shape == (6, 64)
scores = torch.randn(2, 8, 5, 5) # (B, h, T, T)
keep = torch.ones(2, 5, dtype=torch.bool)
keep[1, 3:] = False # padding at the end of sequence 2
mask = keep[:, None, None, :] # (2, 1, 1, 5)
masked = scores.masked_fill(~mask, float("-inf"))Determinant, rank, inverse and solve in NumPy, run once on fixed matrices.
import numpy as np
corr = np.array([[1.0, 0.61], [0.61, 1.0]]) # face pixels
lora = np.array([[2.0, 2.0], [1.0, 1.0]]) # rank-1 update
w_mlp = np.array([[-1.0, 1.0], [1.0, -1.0]]) # the MLP walkthrough's layer
dets = [np.linalg.det(mat) for mat in (corr, lora, w_mlp)] # 0.628, 0, 0
ranks = [np.linalg.matrix_rank(mat) for mat in (corr, lora, w_mlp)] # 2, 1, 1
inv_corr = np.linalg.inv(corr) # [[1.593 -0.971] [-0.971 1.593]]
z = np.linalg.solve(corr, np.array([1.0, 1.0])) # [0.621 0.621]Four traps catch most shape and matrix bugs.
nn.Linear stores (out, in), so a product by hand needs W transposed, W.T.F.normalize (from torch.nn.functional, PyTorch’s scale to length 1).np.linalg.solve skips the error that forming the inverse adds, and np.linalg.lstsq (least squares) does the same for a matrix that is not square. With near-copy features the matrix is close to having no inverse, and the solved weights swing either way. Its condition number, how much a small error in the data can grow in the answer, measures this; its determinant does not.Every vector on this page held fixed numbers. How do we describe numbers that chance picks, like the next token a language model samples? Probability and Bayes Rule starts there.
torch.nn.Linear, broadcasting semantics, clip_grad_norm_.numpy.linalg.solve.