Platform / Data
verifiedData Versioning
Being able to name the exact data a run used, and get it back. Code has had this for fifty years and data mostly has not — a training set is a path on a filesystem that somebody has since appended to, and "the model trained on the March data" is a sentence with no way to check it.
The distinction from Data Provenance is the question being answered. Provenance asks where a document came from; versioning asks which exact set was used, and lets you retrieve it. The implementations either copy, which is exact and expensive, or record content hashes and rely on the underlying storage being immutable, which is cheap and fails silently if anything is ever overwritten in place. Which one you have determines whether a reproduction is a guarantee or a hope.
Storage cost is the objection and it is smaller than it looks, because versions overlap almost entirely. Content-addressed storage keeps one copy of each distinct chunk, so the marginal cost of a version is the size of what changed rather than the size of the dataset — appending a week to a year of data costs a week. The cost that is real is the metadata and the discipline, not the bytes.
shared-chunks holds 50% of the budget; rest holds the remaining 50%.
Data a new version shares with the last one, against what actually changed, in equal units. Drag the shared portion up to watch a version become nearly free — content addressing stores the overlap once, so the cost is the delta and not the dataset.
Reviewed by opendroid · 2026-08-18
- arXiv:2205.02302 — Machine Learning Operations (MLOps): Overview, Definition, and Architecture
- arXiv:2011.09926 — Challenges in Deploying Machine Learning: a Survey of Case Studies