Contrastive Learning

Instead of predicting a label, learn by telling matched pairs apart from mismatched ones. Pull the representations of things that belong together closer, push everything else away. No annotation is needed — the pairing itself is the supervision, and it can come from a caption, a crop, or the next second of audio.

Score every pair in a batch and treat the diagonal as correct, so a batch of size B gives B positives and B²−B negatives for free. Batch size is therefore not a tuning knob but part of the objective: more negatives sharpen the task, which is why these methods are trained at batch sizes that need distributed accumulation.

InfoNCE: ℓ = −log[exp(s(x,y⁺)/τ) / Σ sub _j  exp(s(x,y sub _j )/τ)], a softmax cross-entropy over one positive and the batch's negatives. It lower-bounds the mutual information between the paired views, and the bound is capped by log B — so a small batch limits what the objective can express regardless of the model.