Data Augmentation

Make more training data out of the data you have, by changing it in ways that do not change the answer. A cat flipped left-to-right is still a cat, so a flipped image is a free extra example. It is the cheapest regularisation there is and for a long time it was the difference between a vision model that generalised and one that memorised.

Every transformation encodes an invariance you are asserting, and the assertion can be wrong. Horizontal flips are safe for cats and destroy handwritten digits, where a mirrored 2 is not a 2. Rotation is safe for satellite imagery and not for road signs. Learned augmentation policies search the space of transformations instead of guessing, which works and mostly discovers what a domain expert would have chosen. Interpolating between examples is the odd one out — it augments the labels too, and there is no invariance being asserted at all.

Augmentation adds examples and not information: the extra ones are deterministic functions of the originals, so the effective sample size grows far more slowly than the count does. What it does buy is the invariance itself, which the model would otherwise have to learn from data — so its value is largest where the invariance is real and the data is small, and it falls toward nothing as the corpus grows past the point where that invariance was the binding constraint.