Lottery Ticket Hypothesis

Inside a randomly initialised network there may already be a small subnetwork that, trained alone from those same initial values, matches the full network's accuracy. Training the big network is then partly a search for that subnetwork. The claim is striking and the evidence is real; how far it extends is still argued.

Finding a ticket requires training the full network first, so it is not a route to cheaper training — which is the misreading to guard against. At larger scales the original recipe stops working and needs rewinding to an early checkpoint rather than to initialisation, which weakens the claim in an interesting direction: what matters may be the first few hundred steps rather than the initial draw.

Train, prune the smallest weights, reset the survivors to their values at step 0 (or at an early step k), retrain. The finding is that this beats the same architecture initialised freshly at the same sparsity — so the mask and the initialisation are coupled, and neither alone accounts for the result.