Variational Autoencoder

An ordinary autoencoder learns a latent it can decode, with no promise that the space between two latents decodes to anything sensible. A VAE trains the encoder to emit a distribution rather than a point and pushes those distributions toward a shared prior — so the latent space becomes something you can sample from, not just look up.

The encoder outputs a mean and a variance, a sample is drawn, and the reparameterisation trick keeps that sample differentiable. Two terms compete: reconstruction wants an informative latent, the KL term wants it to look like the prior. Weight the KL too heavily and the decoder learns to ignore the latent altogether, which is posterior collapse and the failure everyone meets first.

Maximise the ELBO, E sub _q [log p(x|z)] − KL(q(z|x) ‖ p(z)), a lower bound on log p(x). The reparameterisation z = μ + σ·ε with ε ~ N(0,I) moves the sampling off the gradient path so the encoder can be trained at all. Scaling the KL by β trades reconstruction against how closely the aggregate posterior matches the prior; at β = 0 the KL term vanishes but the sampling does not, which leaves a noisy autoencoder rather than an ordinary one.