Generative / Objectives
verifiedScore Matching
You can describe a distribution by where its density is going up rather than by the density itself — the gradient of the log-probability, which is called the score. Learning that gradient avoids the normalising constant entirely, and once you have it you can sample by climbing it while adding noise. This is the machinery underneath diffusion.
The trick that makes it practical is training on noisy data at many noise levels: the score is hopeless to estimate where there is no data, and noise fills the space in so the model has a target everywhere. Large noise gives coarse structure and small noise gives detail, which is why sampling anneals from one to the other rather than running at a single level.
The score is ∇ sub x log p(x), and it does not involve the partition function since ∇ sub x log Z = 0. Denoising score matching gives the tractable objective E‖s sub θ (x̃) − ∇ sub x ̃ log q(x̃|x)‖², whose target for Gaussian noise is just −(x̃ − x)/σ² — so predicting the score and predicting the noise added are the same task written twice.
16 values. The left group decays steeply; the right group is 36% of the way to flat, and reads flatter than the left.
The data distribution as the model sees it at one noise level, sharp and smoothed. Drag the noise up to watch structure flatten away — the large-noise end is easy to learn and says little, which is why sampling starts there and anneals down.
Reviewed by opendroid · 2026-08-18
- arXiv:1907.05600 — Generative Modeling by Estimating Gradients of the Data Distribution
- arXiv:2011.13456 — Score-Based Generative Modeling through Stochastic Differential Equations