Multimodal / Generation
verifiedVideo Generation
Generating a video is not generating many images. Each frame has to be plausible and the sequence has to be consistent — the same person, the same room, objects that move the way objects move. A model that produces excellent frames independently produces a flickering mess, and fixing that is the entire problem.
The approach that works is to take an image model and add temporal layers, then fine-tune on video, which reuses everything the image model learned about appearance and spends the new capacity on motion. Latent-space generation is essential rather than optional here: at pixel resolution the tensors are too large to attend over, so the compression that made image diffusion practical is what makes video diffusion possible at all.
The cost scales with frames and with the temporal attention between them, so a clip is far more than its frame count times an image. Consistency is the term that does not amortise — checking that frame fifty matches frame one requires attention across the whole clip, which is quadratic in length, so doubling the duration more than doubles the work. That is why generated clips are short, and why extending them is done by conditioning on the previous clip rather than by generating longer.
temporal-attention holds 33% of the budget; rest holds the remaining 67%.
Compute spent relating frames to each other, against the compute of generating them independently, in equal units. Drag the temporal attention up to watch consistency take the budget — it is quadratic in duration, which is why clips are short and extended by conditioning.
Reviewed by opendroid · 2026-08-18
- arXiv:2311.15127 — Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- arXiv:2304.08818 — Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models