Optical Flow

For every pixel in one frame, where did it go in the next? The answer is a dense field of little arrows, and it is the raw material for tracking, video compression, frame interpolation and stabilisation. It sounds like a matching problem and it is, but with a catch: a pixel on a smooth surface looks exactly like its neighbours, so matching alone cannot decide.

Modern methods build a correlation volume — every pixel of the first frame scored against candidate positions in the second — and then iterate a refinement over it rather than reading off the best match. Iteration matters because the first estimate is unreliable in exactly the regions where the appearance is ambiguous, and a few passes of refinement propagate confident matches from textured regions into flat ones.

Brightness constancy gives one equation per pixel and the flow has two components, so the problem is underdetermined at every single pixel — the aperture problem, and it is a statement about the data rather than about any method. Every approach therefore imposes a second constraint, historically a smoothness prior and now a learned one. The correlation volume's cost is the other half of the arithmetic: it grows as the square of the search radius, which is why nobody searches wide and everybody searches coarse-to-fine.