Building Reproducible ML Processes with an Open Source Stack
summary
Claims a reproducible ML experiment needs three concrete pieces: code (Git + KubeFlow), data (MinIO + lakeFS) and environment (infrastructure-as-code), and shows a working demo that recreates an experiment by snapshotting exact input files, code commit IDs and environment. lakeFS provides file-level versioning via immutable object storage pointers, branches, commits, local clones and mounts, pre-merge hooks and garbage collection; it supports Iceberg/Delta/Hudi, has GDPR delete work in progress, and is used end-to-end to roll back bad data.
AI-written summary · about
chapters (8)
- 0:00Reproducibility in ML pipelines
- 1:24Manual data versioning frustrations
- 5:02Three pillars of reproducible workflows
- 8:56Why data version control matters
- 10:00LakeFS data version control
- 14:18Local work with LakeFS
- 23:29Benefits of data version control
- 26:42Q&A
related talks
32mReproducibility and versioning of ML systems
31mGetting ML Right in a Complex Data World
33mCompliant Data Management and Machine Learning on Kubernetes
26mCI/CD For Data – Building Data Development Environments With Open Source Stacks
30mReproducible Machine Learning
29mReproducibility in ML Systems: A Netflix Original
29mML Model Traceability and Reproducibility by Design
01h 25mBest practices and solution design using data versioning for machine learning