Image Generation Evaluation

How do you score a picture nobody asked for and nothing can mark? There is no reference to compare against, so the standard answer compares distributions instead: generate a lot of images and ask whether the set looks like the set of real ones. That works better than it sounds and is blind to things a person notices immediately.

The dominant metric embeds both sets with a fixed pretrained network and compares the two Gaussians fitted to those features. It is sensitive to blur, to artefacts, and to a generator that ignores part of the distribution — but it says nothing about whether an individual image is good, nothing about whether it matches its prompt, and it moves when the embedding network changes. Prompt fidelity needs a separate measure, and both keep getting overruled by human preference studies.

Comparing distributions rather than samples has a specific consequence: a generator that memorises the training set scores perfectly. The metric asks whether the generated distribution matches, and a copy matches exactly, so novelty is not measured at all by construction. The estimate is also biased by the sample count, approaching its true value from above as the count rises, so two numbers computed at different sample sizes are not comparable — and papers report the count for exactly that reason.