KL Divergence

How much do you lose by believing the wrong distribution? KL divergence answers in bits: the extra length of every message when you encode reality's symbols using your model's code. It is zero only when the two agree, and it is not symmetric — being wrong about a rare event costs differently depending on which way round you are wrong.

The asymmetry is a design decision, not a wart. Forward KL, averaging over the data, punishes a model that puts no mass where data is and so spreads out to cover everything; reverse KL, averaging over the model, punishes putting mass where data is not and so collapses onto one mode. Variational inference minimises the reverse, which is why a VAE's posterior is famously over-confident, and RLHF penalises the reverse to keep a policy near its reference.

D sub _KL (p‖q) = Σ sub _x  p(x)·log(p(x)/q(x)) = H(p,q) − H(p) ≥ 0, with equality only when p = q. It is not a metric: it fails symmetry and the triangle inequality both, which is why the symmetrised Jensen–Shannon divergence exists and why calling it a distance causes trouble.