Implicit Regularization

A network with far more parameters than data can fit that data in many ways, most of which generalise terribly. Gradient descent reliably finds one that does not — without anyone telling it to. The optimiser is choosing among solutions, and the choice is a form of regularisation nobody wrote down.

This is why the classical parameter-counting account of overfitting fails on modern networks and why explicit regularisation is often unnecessary. It also means the optimiser is a modelling decision rather than an implementation detail: changing from SGD to full-batch, or changing the learning rate, changes which solution you get among many with identical training loss.

On separable data with a linear model, gradient descent on the logistic loss converges in direction to the maximum-margin solution — no margin term appears in the objective anywhere. The bias is a property of the optimisation path rather than of the loss, which is why two optimisers minimising the same objective to the same value can generalise differently.