Deduplication

Web corpora contain the same text many times — mirrored pages, quoted passages, licence boilerplate. Removing duplicates makes models better and training cheaper at once, which is rare enough to be worth noticing. It also removes most of what a model would otherwise memorise verbatim.

Exact matching is not enough; near-duplicate detection with MinHash over shingles is the standard tool, and the threshold is a real decision rather than a default. The strongest argument for doing it is not the quality gain but the memorisation one — extraction risk scales with how often a sequence appeared, so removing duplicates removes attack surface faster than it removes data.

Estimate Jaccard similarity between documents by MinHash signatures and bucket with LSH, giving approximate near-duplicate detection in roughly linear time rather than the quadratic cost of all pairs. Since recoverability of a sequence rises steeply with its duplicate count, deleting the tail of repeats removes a disproportionate share of the memorisation risk.