Bayesian Neural Network

Instead of one number per weight, keep a distribution over each. Predicting means sampling several networks from that distribution and looking at how much they disagree — so uncertainty is not bolted on afterwards, it falls out of the same object that makes the prediction. The idea is older than deep learning and has never quite become practical at scale.

Exact inference is hopeless, so everything shipped is an approximation: variational families that assume the weights are independent, Laplace approximations around a trained point, or Monte Carlo dropout, which reinterprets a regulariser you were already using as a posterior sample. Each approximation is cheap in a different way and wrong in a different way, and the disagreement between them is why the field has no default.

The posterior can be tempered — raised to a power 1/T — and the empirical finding is awkward: deep networks work better at T well below 1, a sharpened posterior rather than the true one. Raising T flattens the posterior toward the prior and performance degrades, so the Bayes posterior that theory prescribes is not the one that works. Whether that indicts the prior, the likelihood, or the data augmentation is still open.