Meta-Learning

Learn something that makes learning the next thing faster. Rather than fitting one task, fit an initialisation, or an update rule, or a representation, such that a handful of examples is enough afterwards. It is training for adaptability rather than for performance.

MAML's version optimises through the inner update, which needs second derivatives and is expensive enough that first-order approximations are the default in practice. The uncomfortable result is that much of the benefit comes from feature reuse rather than rapid adaptation — the learned initialisation is a good representation, and freezing most of it and training a head recovers most of the gain.

Minimise the loss after one or more gradient steps: min sub _θ  Σ sub _tasks  L(θ − α∇L sub _task (θ)). Differentiating through that inner step gives a Hessian-vector product, which first-order variants drop entirely — and they lose little, which is itself evidence about where the benefit comes from.