Manifold Hypothesis

A 224-pixel colour image lives in a space of roughly 150,000 dimensions, and almost every point in that space is static. Real images occupy a vanishingly thin sheet inside it. The hypothesis is that natural data lies on a low-dimensional surface curved through a high-dimensional space, and that most of learning is the business of finding that surface.

It is why dimensionality reduction works at all, and why the curse of dimensionality is survivable in practice: the effective dimension is far below the ambient one. Measured intrinsic dimensions for image datasets come out in the tens against ambient dimensions in the tens of thousands — and models trained on data of lower intrinsic dimension generalise better, which makes it a prediction rather than a picture.

A d-dimensional manifold embedded in ℝ super ^D  with d ≪ D. Sample complexity scales with d rather than with D, which is what makes learning in high dimensions possible at all: the volume argument that makes ℝ super ^D  hopeless does not apply to a surface of dimension d sitting inside it.