Gradient Descent

The gradient points in the direction the loss increases fastest, so stepping the other way makes it smaller. Repeat a few hundred thousand times. That is the whole algorithm — the sophistication in modern training is entirely in how big the step is and how much of the previous steps it remembers.

Subtract the gradient scaled by a learning rate. On a non-convex surface there is no guarantee of reaching a global minimum and in practice nobody needs one: in a large network the local minima reached sit close together in training loss. How well they generalise is a separate question, and one the batch size turns out to affect. Step size is the parameter that decides whether it converges, oscillates, or diverges.

θ sub _t+1  = θ sub _t  − η∇L(θ sub _t ). For L with L-Lipschitz gradients, convergence to a stationary point requires η < 2/L; above that the iterates diverge. The bound is unknowable in practice, which is why the learning rate is tuned rather than derived.