Sim-to-Real

Simulators are fast, safe and free, and they are all wrong in ways that matter. A policy trained in one usually fails on the physical system it was meant for, because it learned details of the simulation rather than of the task. The counter-intuitive fix is not a better simulator but a deliberately worse one — many bad simulators instead of one good one.

Domain randomisation varies masses, frictions, latencies, textures and lighting so widely that reality looks like one more sample from the training distribution. The trade is directly visible: the wider the randomisation, the more robust and the less optimal the policy, since it must work everywhere rather than well anywhere. Randomising a parameter the real system does not vary buys nothing and costs performance.

Train to maximise the expected return over a distribution of simulator parameters rather than a single setting, which turns the problem into robust optimisation. As the parameter distribution widens the optimal policy converges toward one insensitive to those parameters — that insensitivity is the transfer, and the lost performance at any single setting is what it costs.