Exploration and Exploitation

Take the best action you know about, and you never find the better one you have not tried. Try new things, and you spend the run paying for it. Every agent has to split its behaviour between the two, and no split is right for long — early on almost everything is worth trying, and late almost nothing is.

ε-greedy is the crude version and works surprisingly often; optimistic initialisation, count-based bonuses and curiosity signals are the graded ones. The failure mode that matters is an agent that stops exploring before it has found the reward at all, which in a sparse-reward world is most of them — and a curiosity bonus is a reward the designer never intended to optimise.

With ε-greedy the agent acts randomly a fraction ε of the time, so regret accumulates linearly in ε while the chance of missing a better arm falls with it. Annealing ε turns a fixed trade into a schedule, and the same shape appears in decoding: temperature and top-p are the exploration knob for a language model.