Training Data Licensing

Whether a model may be trained on a piece of text is a legal question, it is answered differently in different jurisdictions, and it is not settled anywhere. What is clear is the engineering consequence: the answer changes what a corpus may contain, and a corpus assembled before anyone asked cannot be un-assembled afterwards.

The responses split three ways. Train on everything and argue the use is permitted — the position most large models were built on, and the one being litigated. Restrict the corpus to permissively-licensed material, which is much smaller and produces measurably weaker models. Or separate the two: train the parametric model on low-risk data and put the risky material in a retrieval datastore that can be edited or removed without retraining, which converts a legal problem into a systems one.

The trade is between corpus size and legal exposure, and it is steep because permissively-licensed text is a small fraction of what exists. Restricting to it costs the model whatever the excluded distribution contained, and that loss does not shrink with scale — a larger permissive corpus is still a corpus missing the same kinds of material. The datastore approach is interesting precisely because it changes the shape of the trade rather than choosing a point on it.