On this page

1 The linear model 2 The mean squared error cost 2.1 The gradient of the cost 2.2 The gradient descent update 2.3 The closed-form solution 3 Gradient descent on a 2D fit 4 Linear regression in practice 5 Failure modes of linear regression 6 Polynomial features and logistic regression 7 References

Linear Regression

Linear regression predicts a number, say a house price, as a weighted sum of features.

Figure 1. Each box's marker is the number of the section that teaches it.

Gradient descent (Section 2.2) repeats one step over the whole data set, about two multiply-adds per feature per sample; the closed form (Section 2.3) solves once.

1 The linear model

For a house the features x₁ and x₂ are its size and bedrooms.

The prediction is a weighted sum of the features plus a constant shift.

ŷ = w₁x₁ + w₂x₂ + b

The prediction ŷ is the sum of each feature's value times its weight, shifted by the bias b. The weights w₁ and w₂, one per feature, say how much each feature moves the prediction, and b, also called the intercept, is the prediction when every feature is zero.

The Linear Regression notebook in the Machine-Learning-Concepts repo generates data from this model: 2,000 samples, weights 5 and 2, bias 2, no noise; its fit adds a small ridge penalty (0.01, Section 4) that the cost below leaves out.

2 The mean squared error cost

Training picks the weights and bias with the lowest cost J, the mean squared error.

The cost scores how wrong the current weights and bias are.

J = 12MΣ_i (ŷ_i − y_i)²

The cost J is half the average, over all M samples, of the squared residual: Σ (sigma) means sum over every sample i, and ŷ_i minus y_i is how far off we are on sample i, prediction minus truth. Squaring makes big misses hurt more than small ones while the sign stops mattering. The half, the 2 in 2M, is there so the slope comes out clean: squaring brings down a factor of 2 when you take the slope, and the half cancels it, which is why the gradient line divides by M alone.

Squared error suits noise that is bell-shaped (Gaussian) around the line; the Likelihood and EM page shows why.

Figure 2. Gradient descent steps down the bowl from 1; one solve jumps to 2.

2.1 The gradient of the cost

This cost is convex: shaped like a bowl, with one minimum and no other. Walking downhill needs the slope, the gradient.

The gradient gives the slope of the cost along each weight.

∂J∂w_j = 1MΣ_i (ŷ_i − y_i) x_ij

For each weight w_j, the slope is the average over samples of (how wrong we are) times (the feature value): the fraction is the partial slope, ∂ (the partial symbol) J over ∂ w_j, how J changes when only that one weight moves, and Σ (sigma) means sum over every sample i. ŷ_i minus y_i is how wrong we are on sample i, and x_ij is that sample's value of feature j. The sign does the steering: a negative residual with a positive feature gives a negative slope, so stepping against it pushes the weight up and shrinks the miss.

2.2 The gradient descent update

Gradient descent steps against the gradient, scaled by the learning rate, and repeats.

Gradient descent steps downhill, then repeats.

w_j ← w_j − lr · ∂J∂w_j

The arrow means "gets replaced by", so w_j becomes itself minus lr, the learning rate, times the slope, the partial slope ∂J over ∂w_j from the gradient line, so every weight steps downhill at the same time, and the same update runs for the bias, with its own gradient. Raise lr and steps grow until they overshoot; lower it and the walk can stop short of the bottom; the notebook uses lr = 0.1 for 1,000 steps.

Gradient descent in NumPy for M rows and 2 features; the snippet and Figure 3 run 200 steps, the notebook 1,000.

import numpy as np
# X: (M, 2), y: (M,)
lr, n_steps = 0.1, 200
w, b = np.zeros(2), 0.0
M = len(y)
for step in range(n_steps):
    resid = X @ w + b - y  # (M,)
    grad_w = X.T @ resid / M
    grad_b = resid.mean()
    w = w - lr * grad_w
    b = b - lr * grad_b

1Sets the step size lr and starts the weights at 0.
2Residuals ŷ − y, the term inside the cost (Section 2).
3Gradients, the card in Section 2.1.
4One step downhill, the card in Section 2.2.

2.3 The closed-form solution

For squared error the bottom can be solved for directly: least squares, with one matrix inverse (the matrix that undoes a multiplication).

The closed form finds the best weights in one solve.

w = (X^TX)⁻¹X^Ty

X is the whole dataset stacked as a matrix, one row per sample, with a column of ones carrying the bias. The raised T is the transpose, X with its rows and columns flipped, y is the targets stacked alongside, and the ⁻¹ is the matrix inverse. One solve replaces hundreds of steps, and the solved w folds the bias in as one of its entries. The form comes from setting every slope of the cost to zero at once: X^TX measures how the features overlap, X^Ty how each feature lines up with the targets, and the inverse undoes the overlap. When two features are near copies, X^TX is close to having no inverse and the solved weights swing wildly.

On clean data this recovers the truth exactly, w₁ = 5, w₂ = 2 and b = 2.

The closed form on the same X and y, solved with lstsq, which stays accurate when features are near copies, unlike the inverse.

import numpy as np
lstsq = np.linalg.lstsq
ones = np.ones(len(X))  # for b
Xb = np.column_stack([ones, X])
sol = lstsq(Xb, y, rcond=None)
b, w1, w2 = sol[0]

1Adds a column of ones so the solve also finds b.
2Solves for every weight at once (Section 2.3).

3 Gradient descent on a 2D fit

Both routes on 300 samples (the notebook uses 2,000).

true target y prediction fitted line where the walk ends

Figure 3. The cost falls from the start 1 to the end of the walk 2 as the fitted line turns; Snap jumps to the closed form.

Learning rate 0.102 Iterations 200

4 Linear regression in practice

The same fit in scikit-learn; R² is the share of the target's variation the line explains.

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_regression(2000, 2)
parts = train_test_split(X, y)
Xtr, Xte, ytr, yte = parts
scaler = StandardScaler()
ols = LinearRegression()
model = make_pipeline(scaler, ols)
model.fit(Xtr, ytr)  # one solve
r2 = model.score(Xte, yte)

1Makes 2,000 two-feature rows and holds out a test split.
2Scales the features, then ordinary least squares.
3Fits in one solve and scores R² on the test split.

Setting	Where it enters	Raise it	Lower it	Default
Learning rate `lr`	Scales each step in the update card	Faster, until steps overshoot	Slower; tiny values stop short	0.1 for 1,000 steps in the notebook
Iterations	Update steps run	Walks closer to the bottom	Stops above it	1,000 notebook, 200 Figure 3
Ridge penalty `α`	Adds `α` times the squared weights to the cost	Shrinks weights toward 0 and calms twin features	At 0 it is plain least squares	1.0 in scikit-learn's Ridge
`eta0`	First step size of `SGDRegressor`	Faster start, can diverge	Slow progress per pass	0.01, shrinking as eta0 / t^0.25 (`learning_rate="invscaling"`)

LinearRegression solves least squares with SciPy's lstsq: time grows with samples times features squared, and memory holds the data once. For data too big for memory, SGDRegressor takes the steps above in batches through partial_fit.

5 Failure modes of linear regression

A far point's cost grows with the square of its miss, so the fit bends toward it.

Figure 4. One far point 1 tilts the fit from slope 5.4 to 3.7; without it the fit follows the rest.

Cause	Symptom	Fix
A far point	The line tilts toward one sample	Check residuals; use a robust loss such as Huber, which grows only linearly for big misses
Unscaled features	Gradient descent zigzags and needs a tiny learning rate	Standardize each feature first, shifting it to mean 0 and spread 1 (Gradient Descent)
Two features that move together	Their weights can trade off wildly while their combined effect stays put	Drop one, or add an L2 penalty (Regularization)
Predicting past the data	The line keeps going where no sample was seen	Trust it only inside the range it was fit on

6 Polynomial features and logistic regression

Add squared features and the same code fits a polynomial curve.

Pass the line through the sigmoid, which maps any number into 0 to 1, and it gives a probability: logistic regression.

7 References

Legendre, A.-M. (1805). Nouvelles méthodes pour la détermination des orbites des comètes. (the first published account of least squares)
Hastie, T., et al. (2009). The Elements of Statistical Learning, Chapter 3 (Linear Methods for Regression). (least squares and the closed-form solution)