Self-Supervised Learning

Supervised learning needs labels, and labels are expensive. Self-supervised learning makes the labels out of the data itself: hide part of it and predict the hidden part. No annotation is involved anywhere, so the supply of training signal is bounded only by how much data exists.

The pretext task is the whole design. Predict the next token, the masked patch, or which augmented view came from which image — each teaches something different, and a task that is too easy teaches nothing while one that is impossible teaches noise. It is the reason pretraining works, and the reason "unsupervised" is a slightly wrong name for what modern pretraining actually does.

The objective is supervised in form and unsupervised only in provenance: the labels are a deterministic function of x rather than of an annotator, so the usual machinery applies unchanged. What differs is that the label distribution is chosen by whoever designed the pretext task, which makes the inductive bias an authored quantity rather than a property of the data.