the
∞
.ai
Search
/
Concepts
trail ·
0 nodes
Every concept
Information
8 concepts, 8 reviewed and merged.
Compression
Rate-Distortion
Lossless compression has a floor.
verified
Divergence
KL Divergence
How much do you lose by believing the wrong distribution? KL divergence answers in bits: the extra length of every…
verified
Foundations
Entropy
How surprised should you expect to be? Entropy measures that in bits: a fair coin costs one bit per flip, a coin that…
verified
Mutual Information
How much does knowing one thing tell you about another? Mutual information measures the reduction in surprise about Y…
verified
Generalization
Minimum Description Length
The best model of some data is the one that lets you write the data down in the fewest bits — counting the model…
verified
Methods
Noise-Contrastive Estimation
Fitting a probability distribution normally means computing a sum over every possible outcome, which is hopeless when…
verified
Objectives
ELBO
The quantity you want — how likely your data is under the model — is an intractable integral.
verified
Representation
Information Bottleneck
A good representation keeps what is useful and throws away everything else.
verified