Likelihood vs Sample Quality

A model can assign excellent probability to the data and produce terrible samples, or produce beautiful samples while assigning the data almost no probability. The two are close to independent in high dimensions, which means a paper reporting one has told you little about the other.

This is why generative evaluation is a mess rather than merely difficult. FID measures distance between feature statistics and is sensitive to the feature extractor and the sample count; likelihood is measured in bits per dimension and is dominated by imperceptible low-level structure. Reporting both, with sample counts and extractors stated, is the minimum — and neither answers whether the model covers the data.

In high dimensions log-likelihood is dominated by the many dimensions carrying little perceptual content, so a model can gain bits by fitting texture noise while losing nothing measurable in samples. Conversely a mixture placing tiny mass on the data and the rest on good-looking samples scores well perceptually and arbitrarily badly in likelihood — the two orderings can be made to disagree by construction.