Information Bottleneck

A good representation keeps what is useful and throws away everything else. The information bottleneck states that as an objective: squeeze the input through a channel that keeps as little of it as possible while still predicting the label. Compression is not a side effect here — it is what the representation is for.

The claim that made it famous in deep learning is that training has two phases: a short fitting phase and a long compression phase where representations discard input information while holding accuracy. That claim is contested — later work found the compression phase depends on the activation function and on how mutual information was estimated in a deterministic network, where it is strictly speaking infinite. The objective remains useful; the empirical story about training dynamics did not survive replication cleanly.

Minimise I(X;T) − β·I(T;Y) over encoders T of X. β sets the exchange rate between compression and prediction: at β = 0 the best T is constant, and as β grows T is allowed to keep more of X in order to say more about Y. The frontier this traces is the same curve rate-distortion theory draws, with prediction in place of fidelity.