Backpropagation

To improve, a model needs to know how much each of its millions of parameters contributed to being wrong. Backpropagation computes that by running the chain rule backwards through the network, reusing each layer's result for the layer before it. Without the reuse the calculation would be hopeless; with it, one backward pass costs about what the forward pass did.

Reverse-mode automatic differentiation over the computation graph. The forward pass stores activations because the backward pass needs them, which is why memory scales with depth and why checkpointing trades recomputation for it. The cost is roughly two forward passes, independent of parameter count.

For y = f sub _L (…f sub ₁ (x)), ∂L/∂θ sub _i  = (∂L/∂y)·Π sub _j>i (∂f sub _j /∂f sub _j−1 )·(∂f sub _i /∂θ sub _i ). Reverse mode evaluates the product right to left so each factor is applied to a vector rather than a matrix, giving cost proportional to outputs rather than to parameters.