Video Understanding

A video is a stack of images, but treating it as one wastes almost everything: consecutive frames are nearly identical, so most of that data is redundant, while the thing you actually want — what is happening — lives in the differences. Recognising an action needs motion at a fine time resolution and appearance at a fine spatial one, and those are different requirements.

The design that made this concrete runs two pathways at different frame rates: a slow one, sampling few frames but with many channels, which carries what things look like; and a fast one, sampling many frames with few channels, which carries how they move. Lateral connections let them share. It works because the two kinds of information genuinely have different sampling requirements, so spending equally on both was always the wrong split.

The fast pathway can be cheap because channels and frame rate multiply. Convolution cost grows with the square of the channel count and only linearly with the frame count, so running the fast pathway at eight times the frame rate with an eighth of the channels costs about an eighth of the slow one. Temporal resolution is bought with channel width rather than with compute. That trade only works because motion is low-dimensional compared with appearance — you need many samples of it but not many features per sample, which is the observation the whole architecture rests on.