Elastic Weight Consolidation

If some weights mattered a lot for the old task, make them expensive to move. EWC estimates that importance and adds a spring pulling each weight back toward its old value, stiff where the weight mattered and slack where it did not. The model can still learn — it just has to route around what it needs to keep.

Importance is estimated from the diagonal of the Fisher information, which is cheap and assumes the weights are independent — an assumption that is false and is why the method degrades over long task sequences. Penalties also accumulate: after many tasks the model is stiff everywhere and stops learning, which is the stability-plasticity trade appearing as a concrete failure rather than a principle.

Add Σ sub _i  (λ/2)·F sub _i ·(θ sub _i  − θ* sub _i )² where F sub _i  is the Fisher diagonal at the old solution. This is a Laplace approximation to the old task's posterior — a Gaussian centred on θ* with precision F — so EWC is doing Bayesian updating with a very rough posterior, and the roughness is the diagonal assumption.