GAN

Two networks against each other: one makes samples, the other tries to tell them from real data, and the first improves by fooling the second. There is no likelihood anywhere in it — the objective is a game, not a fit — which is why GANs produced sharp images years before likelihood models did, and why training them was so unreliable.

The instability is structural rather than a tuning problem. The generator's gradient comes from a discriminator that is itself moving, so there is no fixed objective descending; a discriminator that wins too completely gives vanishing gradients, and one that loses gives useless ones. Wasserstein losses and gradient penalties made it survivable, and a large study found most reported architectural gains disappeared under equal tuning budgets.

min sub _G  max sub _D  E sub _x [log D(x)] + E sub _z [log(1 − D(G(z)))]. At the optimal discriminator the generator minimises the Jensen–Shannon divergence, which saturates when the supports do not overlap — the reason early training stalls, and the reason Wasserstein distance, which stays informative for disjoint supports, replaced it.