Landscape / Foundations
verifiedSharpness
Two solutions with the same training loss can sit in very different neighbourhoods: one in a narrow valley where moving slightly costs a lot, another on a wide plateau where it barely matters. The wide one usually generalises better, and the reason is intuitive — a test distribution shifts the surface slightly, and a flat solution survives the shift.
Large-batch training tends toward sharper solutions, which is a large part of why it generalises worse at matched steps, and sharpness-aware minimisation attacks that directly by optimising the worst loss in a neighbourhood rather than the loss at a point. The caveat that matters: sharpness is not reparameterisation-invariant, so a network can be made to look flat by rescaling weights without changing the function.
Sharpness is read off the Hessian's eigenvalue spectrum — its largest eigenvalue, or its trace. SAM minimises max over ‖ε‖ ≤ ρ of L(θ + ε), approximated by one ascent step and one descent step, which costs a doubled backward pass. The spectrum is what the figure here draws: a sharp minimum has a few enormous eigenvalues and a bulk near zero.
16 values. The left group decays steeply; the right group is 36% of the way to flat, and reads flatter than the left.
The Hessian's eigenvalues at a solution, as found and as flatness is sought. Drag the flatness up to watch the few enormous directions come down toward the bulk — that spread is what sharpness measures.
Reviewed by opendroid · 2026-08-18
- arXiv:2010.01412 — Sharpness-Aware Minimization for Efficiently Improving Generalization
- arXiv:1609.04836 — On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima