Linear Regression

Fit a straight line through points, choosing the one whose distances to them are smallest overall. It is the simplest thing that counts as learning, and nearly everything since is a variation on the same three parts: a parameterised function, a loss saying how wrong it is, and a procedure for making that smaller.

Least squares has a closed form, which is exactly what makes it the useful base case — the answer exists without any optimisation at all. Everything after it gives up that closed form for a reason: nonlinearity, scale, or a loss whose minimum is not a solvable equation. The vocabulary established here is the vocabulary used everywhere else.

Minimise ‖Xw − y‖² over w, solved by w = (XᵀX)⁻¹Xᵀy whenever XᵀX is invertible. It stops being invertible once features outnumber examples, which is the overparameterised regime — and the regime where the classical story that more parameters means worse generalisation stops holding.