Audio-Visual Learning

Video comes with its own soundtrack, and the two are aligned for free — nobody had to label anything for the audio at a moment to correspond to the pixels at that moment. That correspondence is supervision lying around in enormous quantity, and it is what makes audio-visual pretraining possible without annotation.

The standard objective is correspondence: does this audio clip belong to this video clip. Negatives are other clips, so the difficulty is set by how they are chosen — clips from different videos are easy, clips from the same video at a different time are hard, and the hard ones are where the useful representation comes from. The failure to watch for is a shortcut, where the model learns to match on recording conditions rather than on content.

The supervision is free but not unlimited, because much of a video's audio is uninformative about its pixels — a music track, a voiceover, ambient noise. So the share of a corpus where the correspondence actually carries signal is well below the share where it exists, and it is that first number that bounds what the objective can teach. A larger video corpus with the same proportion of ambient soundtracks adds hours and not much supervision.