Safety Envelope

A learned policy on real hardware can hurt somebody, so it does not get the final say. A safety envelope is a separately-specified region the system may operate in, enforced by something simpler than the policy — a force limit, a workspace boundary, a velocity cap — that can veto any command.

The point of keeping the monitor simple is that it must be verifiable by inspection, which a learned policy is not. This is the same argument Sandboxing makes for agents, with the difference that the consequences are physical and the monitor has to run in the control loop rather than around it. A monitor that is itself learned inherits the failure modes it was meant to catch.

Define a set of admissible states and a controller that keeps the system inside it from any state within; the learned policy proposes and the filter projects onto what is admissible. The guarantee is conditional on the dynamics model used to define the set, so the envelope is only as sound as the model — which is why the sets are conservative rather than tight.