Offload

Keep part of the model somewhere cheaper than the accelerator — host memory, or an SSD — and move it in when it is needed. It trades speed for capacity, and it is what makes a model that does not fit trainable on hardware that cannot hold it.

Optimiser state is the usual thing to move, because it is the largest and it is touched once per step rather than twice like the parameters. Pushing further to NVMe extends this to models far beyond aggregate accelerator memory, at the cost of a much slower link. The engineering that makes either usable is overlap: prefetch the next piece while computing on the current one, so the transfer hides behind arithmetic rather than adding to it — and where it cannot be hidden, the accelerator sits idle waiting for a disk.

The trade is bandwidth against capacity. Host memory is roughly an order of magnitude slower to reach than device memory and NVMe another order below that, so a step that would take t on-device takes t plus the transfer time not hidden by overlap. The condition for offload to be free is that arithmetic per byte moved exceeds the ratio of compute throughput to link bandwidth — which is Arithmetic Intensity again, applied to a link nobody thinks of as the bottleneck until it is.