Loss Function

Training needs a single number saying how wrong the model currently is. The loss function produces it. Everything else — gradients, updates, the entire training run — is machinery for pushing that number down, so the choice of what it measures is the choice of what the model becomes good at.

Map a prediction and a target to a scalar, differentiable almost everywhere so a gradient exists. It is a proxy, not the goal: nobody wants low cross-entropy, they want a useful model, and the gap between the two is where most surprising training behaviour lives. Loss going down while the thing you care about stagnates is the normal case, not a bug.

L(θ) = (1/N) Σ sub _i  ℓ(f sub _θ (x sub _i ), y sub _i ) for a per-example loss ℓ. Training seeks argmin sub _θ  L, but only the empirical L over the training set is observable; the quantity that matters is the expectation over the true distribution, which is never computed.