Landscape
8 concepts, 8 reviewed and merged.
Foundations
- Loss LandscapeThe loss is a function of every weight at once, so its surface lives in a space with millions of dimensions.verified
- Mode ConnectivityTwo networks trained from different random seeds end up in what look like separate valleys — but you can often find a…verified
- Saddle PointThe old worry about non-convex optimisation was getting stuck in a bad local minimum.verified
- SharpnessTwo solutions with the same training loss can sit in very different neighbourhoods: one in a narrow valley where…verified