Autoencoder

Train a network to output its own input, but force everything through a narrow layer in the middle. Nothing can pass through unchanged, so the network has to decide what is worth keeping. The bottleneck is the entire idea: without it, copying the input is trivial and nothing has been learned.

An encoder to a latent, a decoder back, and a reconstruction loss. A linear autoencoder under squared error recovers the PCA subspace, so the interesting cases are the nonlinear ones. The failure mode is a latent roomy enough to memorise — capacity has to be constrained by width, noise, sparsity or masking, and which constraint is chosen is what distinguishes the variants from one another.

Minimise ‖x − g(f(x))‖² over encoder f and decoder g. With f and g linear the optimum spans the same subspace as the top principal components, so the nonlinearity is what buys anything at all beyond PCA. Masking a large fraction of the input and reconstructing it is the same objective with the constraint moved off the architecture and onto the data.