Loss Landscape

The loss is a function of every weight at once, so its surface lives in a space with millions of dimensions. Nothing about it can be seen directly, and every picture of it — including the ones in papers — is a two-dimensional slice chosen to make a point. The useful facts about it are statistical rather than visual.

The slices are worth reading with suspicion: a random two-dimensional cut through a high-dimensional surface looks smooth almost regardless of what the surface does, and the well-known visualisations use filter normalisation specifically because unnormalised slices are misleading about scale. What survives scrutiny is measurable — curvature spectra, barrier heights between solutions, how far you can move before the loss rises.

With d parameters, a 2-D slice samples a measure-zero subspace, and the probability that a random plane contains any particular direction of interest is zero. Curvature along a random direction concentrates near the average eigenvalue of the Hessian, which is why random slices look flat while the surface has directions that are enormously sharp.