Latent Diffusion

Running diffusion on raw pixels is enormously expensive, and most pixels carry no information anyone would miss. Latent diffusion compresses the image first with an autoencoder, runs the whole noising and denoising process in that much smaller space, and decodes only at the end. The same model becomes affordable to train and fast enough to serve.

An autoencoder trained once maps images to a latent grid perhaps eight times smaller per side; diffusion then operates entirely there. Text conditioning enters through cross-attention in the denoising network. This is the architecture behind the open image generators, and the compression factor is the single parameter that decides whether it runs on consumer hardware.

A downsampling factor f reduces the spatial grid by f² and the per-step cost by roughly f² — at f = 8 that is sixty-four times less work per denoising step. The autoencoder is trained once and frozen, so the cost is paid once rather than per sample.