Data

8 concepts, 8 reviewed and merged.

Evaluation

Annotation QualityLabels are produced by people who disagree, get tired, and interpret instructions differently.verified
Training Corpus ContaminationEvaluation sets end up in training corpora because both are scraped from the same internet.verified

Foundations

Data Scaling LawsModel quality improves predictably with data, and the improvement is a power law rather than a wall — which is what…verified

Methods

Data FilteringMost of a web crawl is not worth training on — boilerplate, spam, machine translation, pages that are mostly…verified
Data MixingA training corpus is several corpora — web text, code, books, papers, forums — and how much of each you use is a…verified
DeduplicationWeb corpora contain the same text many times — mirrored pages, quoted passages, licence boilerplate.verified
Synthetic DataGenerate training data with a model instead of collecting it.verified

Schedules

CurriculumShow easy examples first and hard ones later, as a teacher would.verified