← All tutorials

On this page

Recommender Systems

A recommender predicts the ratings a user has not given yet and suggests the highest. This page builds matrix factorization on MovieLens ratings, as in the Recommendation notebook.

1 The sparse rating matrix

Each user rated a handful out of thousands of movies, so almost every cell is empty. Recommending means predicting those cells.

2 Matrix factorization

Matrix factorization (Koren et al., 2009) gives every user a vector p_u and every movie a vector q_i of K latent factors, learned numbers with no preset meaning (a larger K fits finer preferences and overfits sooner; K = 20 in the notebook), plus a user bias and a movie bias.

Nothing is hand-picked: training fits all the vectors and biases to the ratings you have by lowering one cost.

predict computes the prediction card's first line for a batch of user and movie pairs; Section 7 trains the tables with the cost J.

def predict(p_u, q_i, b_u, b_i):
    # p_u, q_i: (B, 20)
    dot = (p_u * q_i).sum(dim=1)
    return dot + b_u + b_i

3 Predicting one rating

4 Movie similarity measures

Similarity answers "more like this movie" and covers a new movie with no ratings yet (cold start, Section 8). Rating similarity compares two movie columns of the rating matrix R (zeros where nobody rated). Genre similarity compares two genre vectors from the genre matrix G (1 if the movie has the genre, 0 if not). Both use cosine similarity:

By ratings Toy Story and Toy Story 2 score 0.573; by genres they score 1.0.

import numpy as np

def cosine(a, b):
    na = np.linalg.norm(a)
    nb = np.linalg.norm(b)
    return a @ b / (na * nb)

r_ts = R[:, ts_in_R]
r_ts2 = R[:, ts2_in_R]
rating_sim = cosine(r_ts, r_ts2)
g_ts = G[ts_in_G]
g_ts2 = G[ts2_in_G]
genre_sim = cosine(g_ts, g_ts2)

1. 1Cosine similarity (Section 4).
2. 2Rating similarity.
3. 3Genre similarity.

Rating similarity uses only who rated what, so it is collaborative filtering, recommending from other users' ratings.

Embedding similarity compares each movie's 20-number vector from matrix factorization, its embedding, the same way.

5 Rating neighbors versus genre neighbors

The data are a real slice of MovieLens: 8 movies rated by 6 users.

6 The hybrid recommender

The notebook's hybrid adds one genre signal to the learned score:

genre_cos holds the genre cosine of every movie pair, with each movie's match to itself set to 0.

def hybrid(mf, genre_cos, ids):
    # genre_cos: (movies, movies)
    best = genre_cos[ids].max(1)
    return mf + best.values

The notebook fits the hybrid's model the same way, for 20 full-batch steps.

7 Matrix factorization in practice

The full training loop in PyTorch lowers the cost J from Section 2. The inputs users, items and ratings hold one entry per known rating, with user and movie ids already mapped to rows 0 to n − 1.

from torch import nn
import torch.nn.functional as F
from torch.optim import Adam
from torch.utils.data import DataLoader
from torch.utils.data import TensorDataset

# users, items: LongTensor (N,)
# ratings: FloatTensor (N,)
# 9742 = movies in movies.csv
n_users, n_items = 610, 9742
K, B, lam = 20, 128, 0.0  # no penalty, as in the notebook
P = nn.Embedding(n_users, K)
Q = nn.Embedding(n_items, K)
b_u = nn.Embedding(n_users, 1)
b_i = nn.Embedding(n_items, 1)
parts = [P, Q, b_u, b_i]
mods = nn.ModuleList(parts)
params = mods.parameters()
opt = Adam(params, lr=1e-3)
TD = TensorDataset
data = TD(users, items, ratings)
dl = DataLoader(data, B, True)

def sq(t):
    return t.pow(2).sum()

for epoch in range(50):
    for u, i, r in dl:
        bu = b_u(u).squeeze(1)
        bi = b_i(i).squeeze(1)
        dot = (P(u) * Q(i)).sum(1)
        pred = dot + bu + bi
        mse = F.mse_loss(pred, r)
        pen = sq(bu) + sq(bi)
        pen = pen + sq(P(u))
        pen = pen + sq(Q(i))
        n = len(r)
        loss = mse + lam * pen / n
        opt.zero_grad()
        loss.backward()
        opt.step()

1. 1Vector and bias tables with K = 20, and Adam over all four.
2. 2Shuffled batches of 128 known ratings.
3. 3Prediction card in Section 2, then the mean squared error, cost line 1 divided by the batch size.
4. 4Penalty λ, cost lines 2 and 3, divided by the batch size to match the mean; Surprise uses λ = 0.02.
5. 5One Adam step per batch.

The notebook takes one Adam step (gradient descent with a per-weight step size) per batch of 128 ratings, for 50 epochs, with λ = 0. A step costs about twice a prediction (scores, then gradients) for each rating in the batch; once trained, one prediction is one dot product of length K.

Recommending scores every movie for one user with the prediction card, hides the ones already rated, and returns the k highest. It reuses the trained tables above; R is the rating table with rows and columns in the same order, 0 where unrated.

import numpy as np

def arr(emb):
    w = emb.weight.detach()
    return w.numpy()

Pn, Qn = arr(P), arr(Q)
bu = arr(b_u)[:, 0]
bi = arr(b_i)[:, 0]
u, k = 607, 5  # userId 608
s = Qn @ Pn[u] + bu[u] + bi
seen = R[u] > 0
s = np.where(seen, -np.inf, s)
top_k = np.argsort(-s)[:k]

1. 1Trained tables as NumPy arrays.
2. 2Predicted rating for every movie, prediction card in Section 2.
3. 3Hides movies the user already rated.
4. 4Top k predictions.

Adam keeps two running averages per weight, three copies of every vector in all. Surprise, a Python library for recommenders, ships the same model under the name SVD (a historical name; it is this factorization, not the SVD of the PCA page) and trains it by plain SGD:

Hide some known ratings from J, fit on the rest, and score the hidden ones: they stand in for the empty cells you will actually recommend. A model that fits the rated cells but misses the hidden ones has overfit, memorizing its training ratings instead of learning tastes that carry over. Split ratings by user or time, not random rows only, or the score flatters the model. If you use AdamW instead, its weight decay (default 0.01) shrinks every row each step, even rows of users absent from the batch; the penalty in the loop above touches only the batch's rows.

Take the 48 real ratings of Section 5, hide 10 scattered across users and movies, the easy kind of split, and fit the other 38 by plain gradient descent on J, all 38 in every step, with μ in the prediction and K = 2, so each vector is two numbers, an arrow in a plane. The miss on a set of ratings is their typical gap in stars: square the gaps between predicted and real, average them, and take the square root.

8 Strengths and limits of matrix factorization

Learned vectors need plenty of ratings per user and per movie; three gaps have known fixes.

Every row of the predicted table is a mix of just K movie-vector patterns, the same idea as the eigenfaces in PCA.

9 References

1. Koren, Y., et al. (2009). Matrix Factorization Techniques for Recommender Systems. IEEE Computer, 42(8), 30-37.