On this page
On the regression page a line predicts a number. Here the answer is yes or no: is this iris flower of the species virginica? (The Iris data set holds 150 flowers of three species, four measurements each.) Logistic regression keeps the line and maps its score to a probability with the sigmoid.
Gradient descent (Section 3) repeats one step on all the samples (the notebook runs 5,000 steps at learning rate 1); each step costs about twice a prediction, the scores and sigmoids and then the gradient.
Fit a line to 0-or-1 labels and it predicts values like 2.4 or −0.7, which are not probabilities, and one far point drags the whole line sideways.
So we keep the line as a raw score and pass it through the sigmoid, a function that maps any number into the range 0 to 1. That number reads as a probability, and 0.5 becomes the decision boundary.
The line's raw score can be anything from minus infinity to plus infinity. The sigmoid maps it into the range 0 to 1:
Squared error asks "how far off was the number?" For probabilities the better question is "how surprised were you?" If the truth is yes and you said 90% yes, no surprise. If you said 10% yes, huge surprise. That is the binary cross-entropy cost (the cross-entropy loss with two classes): each sample's cost depends only on the probability it gave its true label.
Cross-entropy is the negative log likelihood of a coin-flip model; the Likelihood and EM page shows why.
The gradient, the arrow that points uphill on the cost so the learning step can go the other way and lower it, has a shape you already know:
One full-batch training step in NumPy, with labels 0 or 1.
import numpy as np
# X: (n, 2), y: (n,) of 0/1
EPS = 1e-12 # keeps log finite
z = X @ w + b
p = 1 / (1 + np.exp(-z))
p = np.clip(p, EPS, 1 - EPS)
yes = y * np.log(p)
no = (1 - y) * np.log(1 - p)
loss = -np.mean(yes + no)
n = len(y)
grad_w = X.T @ (p - y) / n
grad_b = np.mean(p - y)
w = w - lr * grad_w
b = b - lr * grad_bFigure 3. One virginica flower is scored, called at 0.5 and used for one learning step.
lr in the code). Here y = 1, so p − y is negative and each weight grows along its feature; the bias takes the same step with x set to 1.The sigmoid hits one half exactly when the raw score is zero.
Any other threshold t also gives a straight boundary, just shifted:
Every point whose probability clears t lands on the yes side. Raising t calls fewer flowers yes (fewer false alarms, more misses); lowering it calls more. For virginica, t = 0.1 also calls 15 of the 50 versicolor yes, and t = 0.9 keeps only 41 of the 50 virginica (counts from the 800-step fit in Figure 4).
For more than two classes, one-vs-rest runs one yes-or-no model per class (three for Iris). The softmax, the multi-class form of the sigmoid, scores all classes at once:
The Logistic Regression notebook in the Machine-Learning-Concepts repo runs this on the 150 Iris flowers: two petal measurements, centered on the median and scaled to unit spread, and one yes-or-no classifier per class.
One-vs-rest in scikit-learn; StandardScaler centers on the mean rather than the notebook's median.
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
petals = X[:, 2:4] # petal size
sc = StandardScaler()
X = sc.fit_transform(petals)
base = LogisticRegression()
ovr = OneVsRestClassifier(base)
ovr.fit(X, y)
virg = ovr.estimators_[2]
p = virg.predict_proba(X)[:, 1]
t = 0.5 # threshold
called_yes = p >= tFigure 4. The solid line is the chosen class's boundary at t; filled flowers are called yes.
Setosa splits off cleanly and virginica nearly so. Versicolor sits between the other two, so its one straight boundary calls only 12 of its 50 flowers yes at t = 0.5 and takes in 16 virginica too.
When a straight line can almost split the classes, training inflates the weights to push probabilities to 1 and 0. Regularization adds a penalty that grows with the weights.
When yes is rare, class weights multiply each yes sample's loss: a larger weight calls more flowers yes, a smaller one fewer, and the default weights every class 1.
Each line builds one model; "balanced" weights each class by the inverse of its share of the samples.
from sklearn.linear_model import LogisticRegression
LR = LogisticRegression
strong = LR(C=0.1) # C = 1/(2λ)
yes_x5 = LR(class_weight={1: 5})
bal = LR(class_weight="balanced")The boundary is still straight. The SVM page trains the same kind of boundary with the hinge loss, a cost that is zero once a point is past the boundary by a margin.
The usual pipeline; predict_proba returns one row per flower and one column per class.
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
split = train_test_split
parts = split(X, y, stratify=y)
Xtr, Xte, ytr, yte = parts
LR = LogisticRegression
clf = LR(C=1.0, max_iter=1000)
sc = StandardScaler()
model = make_pipeline(sc, clf)
model.fit(Xtr, ytr)
proba = model.predict_proba(Xte)
acc = model.score(Xte, yte)max_iter); too few stops early with a warning.| Setting | Where it enters | Raise it | Lower it | Default |
|---|---|---|---|---|
| C = 1/(2λ) | Inverse of the penalty in the regularization card | Weaker penalty; on separable data the weights grow without bound | Stronger penalty; near 0 every probability drifts to 0.5 | 1.0 |
max_iter | Solver iterations | More time to converge | A ConvergenceWarning, weights not yet settled | 100 |
| Threshold t | Calls yes when p ≥ t | Fewer yes calls, fewer false alarms | More yes calls, fewer misses | 0.5 (predict) |
LogisticRegression uses the lbfgs solver with an L2 penalty; each iteration is about one pass over the data. The penalty is on by default, so scale first, or unscaled features are penalized unevenly. With three or more classes it fits one softmax model, not the notebook's one-vs-rest set, so its probabilities can differ slightly. For streaming data, SGDClassifier with the log loss takes the per-batch steps taught above.
| Cause | Symptom | Fix |
|---|---|---|
| Unscaled features | Slow, uneven steps, with one weight racing while the other crawls | Standardize the features first |
| Perfectly separable data | Weights grow without bound and probabilities hit 0 and 1 | L2 penalty or early stopping |
| Rare yes class | Almost everything is called no | Move the threshold or use class weights |
| Class in the middle | One straight boundary cannot surround it (versicolor in the threshold tool) | A model that bends (MLP) |
Versicolor showed one class that a straight line cannot cut off. XOR, two classes on the two diagonals of a square, is the simplest data set with that problem. Stacking lines with bends between them fixes it, on the next page, Multilayer Perceptrons (MLP).