Platform / Practice
verifiedExperiment Tracking
Recording what was run, with what code, on what data, and what came out — automatically, at the moment it happens. Without it, a result from three weeks ago is a number in a chat message that nobody can reproduce, and the honest position on it is that you do not know whether it was real.
What has to be captured is more than the metrics: the commit, the config, the data snapshot, the environment, the random seed, and the hardware, because any of them can be the reason two runs differ. Teams that capture only metrics discover this the first time a result fails to reproduce and there is nothing to compare. The discipline that makes it work is that logging is not optional in the training script — an opt-in tracker records the runs somebody remembered to track, which is a biased sample of them.
The value is in comparison, so it scales with the number of runs and with how long they stay comparable. One run tracked is a record; a hundred tracked identically is a dataset you can ask questions of — which hyperparameter mattered, whether an effect exceeds run-to-run variance. That last question is the one that needs Determinism and Reproducibility, because without repeated runs of the same configuration there is no bar for an improvement to clear.
untracked-runs holds 50% of the budget; rest holds the remaining 50%.
Runs nobody recorded, against the ones in the tracker, in runs. Drag the untracked count up to watch the record become a biased sample — the runs people remember to log are not a random subset of the runs they did.
Reviewed by opendroid · 2026-08-18
- arXiv:2003.12206 — Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
- arXiv:2205.02302 — Machine Learning Operations (MLOps): Overview, Definition, and Architecture