On this page
Support vector machines find the boundary with the most room on both sides.
The linear SVM (Section 2) trains by gradient steps, one per batch of 32 points, about 141 per pass over the data, for 1,000 passes in the svm-pytorch notebook; the kernel SVC (Section 5) is fit once by a dedicated solver instead.
An SVM picks, among the boundaries that separate the classes, the one farthest from the closest points. Two boundaries can both get every training point right; which do you trust on new data? The one with room to spare.
That boundary has the maximum margin, the most room on both sides before it touches a point, so a new point near the edge still lands on the right side.
Those three points are the support vectors: moving any other point leaves the boundary in place.
The decision line is where the score w·x + b is zero, and the margin lines are where it hits +1 and −1:
With no violations allowed, the widest margin is this problem:
Real data overlaps, so training allows violations and adds a cost for each. The SVM is a linear model trained with the hinge loss, which asks whether the model is right with room to spare. The label times the score is positive on the correct side, and past 1 the loss is zero.
The soft margin (Cortes and Vapnik, 1995) swaps the hard rule for that cost on each violation:
C is not a learning rate: a large C puts little weight on keeping w small (weak regularization), a small C much weight. Swap the hinge for the log loss and this is logistic regression.
The svm-pytorch notebook in Machine-Learning-Concepts trains with PyTorch's SGD: one step per batch of 32 points, 1,000 passes over 4,500 training points (about 141,000 steps), learning rate 0.001. It writes the cost as the mean hinge plus 1e-4 times the squared weights, which is J divided by C with C = 5,000. The hinge has a kink, so the step uses the subgradient: a point short of the margin nudges the weights toward its label, and every other point is left alone.
The notebook's training loop in PyTorch, with X the points and labels y of −1 or +1.
import torch
lr, lam = 0.001, 1e-4
epochs, batch = 1000, 32
w = torch.zeros(2)
b = torch.zeros(1)
w.requires_grad_()
b.requires_grad_()
SGD = torch.optim.SGD
opt = SGD([w, b], lr=lr)
for epoch in range(epochs):
order = torch.randperm(len(X))
for idx in order.split(batch):
f = X[idx] @ w + b
gap = 1 - y[idx] * f
hinge = gap.clamp(min=0)
fit = hinge.mean()
l2 = w.dot(w) + b.dot(b)
loss = fit + lam * l2
opt.zero_grad()
loss.backward()
opt.step()Figure 4. Four steps on a toy set: only the points the margin touches set the boundary.
The same J, trained on 400 points in two overlapping clouds.
Figure 5. The margin and the support vectors for one value of C.
A kernel lets the same margin bend around curved data, and two variants reuse it for regression and anomalies.
A lift adds a feature computed from the old ones, here the squared distance from the center.
In practice no one computes the lift: a kernel scores each pair of points as if they had been lifted.
degree), c = 0 (coef0) and γ = "scale".RBF (radial basis function) scores similarity to nearby points, like distance in k-nearest neighbors. The trained w is a weighted sum of the support vectors, so the score w·x is a sum of dot products, and the kernel replaces each one.
Figure 8. Low gamma underfits, near 1 draws one ring, high gamma bends around each point.
RBF is the default for low-dimensional curved data like the rings. The polynomial kernel bends the boundary into smooth curves, not RBF's tight local curves. When features outnumber points, as in text, linear usually wins: a straight cut in a huge space is already flexible.
Support vector regression draws a tube around the predictions and only adds a cost for points outside it. The one-class SVM draws the boundary around normal examples only, and anything outside is an anomaly.
In scikit-learn, SVR's tube half-width epsilon (default 0.1) sets how big a miss is free: wider ignores more, 0 counts every miss. OneClassSVM's nu (default 0.5) caps the share of training points it may call anomalies; it marks new points +1 or −1.
from sklearn.svm import SVC, SVR, OneClassSVM
clf = SVC(kernel="linear", C=1.0)
clf.fit(X, y)
reg = SVR(epsilon=0.1) # rbf
reg.fit(X, y_reg)
det = OneClassSVM(nu=0.05) # rbf
det.fit(X_normal)
flags = det.predict(X_new) # ±1The standard recipe in scikit-learn.
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(1000, noise=0.2)
parts = train_test_split(X, y)
Xtr, Xte, ytr, yte = parts
svc = SVC(C=1.0, gamma="scale")
sc = StandardScaler()
model = make_pipeline(sc, svc)
model.fit(Xtr, ytr)
acc = model.score(Xte, yte)
n_sv = model[-1].n_support_| Setting | Where it enters | Raise it | Lower it | Default |
|---|---|---|---|---|
| C | Price of the hinge term in J | Narrow, strict margin | Wide, forgiving margin | 1.0 in SVC, on the summed hinge (n in J) |
kernel | The similarity K in the decision sum | Linear, polynomial or RBF decides the boundary's shape | "rbf" | |
| γ (gamma) | Scales the squared distance in RBF | Boundary bends around single points | Smooth, nearly straight | "scale", which is 1 / (features × variance) |
n_components (Nystroem) | Landmarks used to approximate K | Closer to the exact kernel, more time and memory | Faster, coarser boundary | 100 |
SVC, built on libsvm, caches kernel values in 200 MB. Its kernel matrix holds K for every pair of points, so its documentation says fit time grows at least with the square of the samples and suggests LinearSVC or SGDClassifier past tens of thousands, possibly after Nystroem. Scale inside the pipeline, since unscaled features break the RBF distance. SVC handles many classes one pair at a time, and scikit-learn 1.9 deprecates probability=True in favor of CalibratedClassifierCV.
scikit-learn's C multiplies the summed hinge, while the J card's C multiplies the mean, so scikit-learn's C equals the page's C divided by n, and its default 1.0 is the page's C = n.
Tuning C and γ together by 5-fold cross-validation (train on four folds, test on the fifth).
import numpy as np
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
sc = StandardScaler()
svc = SVC(kernel="rbf")
model = make_pipeline(sc, svc)
Cs = np.logspace(-2, 3, 6)
gammas = np.logspace(-2, 2, 5)
grid = {}
grid["svc__C"] = Cs
grid["svc__gamma"] = gammas
gs = GridSearchCV(model, grid)
gs.fit(X, y) # 5 folds by default
best = gs.best_params_Nystroem builds features whose dot products approximate the RBF kernel; RBFSampler is the random alternative.
from sklearn.kernel_approximation import Nystroem
from sklearn.pipeline import make_pipeline
from sklearn.svm import LinearSVC
ny = Nystroem(n_components=300)
lin = LinearSVC(C=1.0)
clf = make_pipeline(ny, lin)
clf.fit(X, y)