Logistic Regression

The same idea as fitting a line, except the answer has to be a probability. Squash the linear score through a curve mapping any real number into zero-to-one, and train it to put high probability on the right class. This is the last layer of nearly every classifier ever built, including the one at the end of a transformer.

A linear score followed by a sigmoid or a softmax, trained with cross-entropy. It is convex, so there is one optimum and nothing to worry about in the initialisation — the property almost everything after it gives up. On separable data the weights never converge to a finite point at all: they grow without bound, and it is the direction that converges.

p(y=1|x) = σ(wᵀx + b) with σ(z) = 1/(1 + e super ^−z ), fitted by minimising −Σ log p(y sub _i  | x sub _i ). On separable data gradient descent drives ‖w‖ to infinity while w/‖w‖ converges to the maximum-margin separator — which is why an unregularised logistic model and a hard-margin SVM agree in the limit despite being derived from different arguments.