Training Data Extraction

A language model can be made to emit sequences it memorised verbatim — addresses, keys, phone numbers — by prompting for them and filtering the outputs by how confident the model is. It is a stronger failure than membership inference, because it produces the record rather than a yes-or-no about it.

Memorisation rises with model size, with how often a sequence appeared, and with how unusual it is — a duplicated string is far likelier to be recoverable than a common one. Deduplication is the cheapest mitigation and it works well, which makes it the first thing to check when a model is found to regurgitate. Whether memorisation is a defect depends on the string: the same mechanism stores a quotation and a private key.

Extraction ranks candidate generations by a ratio of the model's likelihood to a reference model's, isolating what this model finds unusually easy rather than what is merely common text. Recoverability scales with duplicate count, so deduplication cuts the attack surface superlinearly relative to the data removed.