Saddle Point

The old worry about non-convex optimisation was getting stuck in a bad local minimum. In high dimensions that worry was misplaced: almost every critical point is a saddle, curving down in some directions and up in others, because a point is a minimum only if all million directions curve up at once.

Saddles slow training rather than stopping it — gradients are small near one and the escape direction exists but is weak — which is a large part of why momentum helps and why plateaus appear in loss curves. This is also why local minima are rarely the explanation for a stalled run, and why looking for one is usually the wrong diagnosis.

At a critical point the Hessian's eigenvalue signs decide the type, and a minimum requires all d to be positive. If signs were independent with probability p of being positive, minima would have probability p super ^d  — vanishing in d — and the loss at the rare true minima concentrates near the global value, which is the second half of the result.