Data Mixing

A training corpus is several corpora — web text, code, books, papers, forums — and how much of each you use is a decision with large effects. Code improves reasoning on non-code tasks; too much of any single source costs breadth. The proportions are tuned like hyperparameters and reported far less often.

Tuning them directly is expensive because each trial is a training run, so the practical methods fit proxy models at small scale and transfer the weights up. That transfer is the assumption to watch: a mixture optimal for a small model is not obviously optimal for a large one, and the literature has examples of the ranking changing with scale.

Sample domain i with weight w sub _i  summing to one; the loss is a weighted sum over domains and the gradient is dominated by whichever has the most mass. Optimising the weights against a target distribution is a bilevel problem — the inner problem is training — which is why the working methods approximate it with small proxy models rather than solving it.