Reproducible Machine Learning
summary
Argues that reproducibility fails from missing data versions, hidden randomness, and untracked experiments and gives concrete fixes: Netguru uses mlflow and Kilt for data and experiment versioning, records seeds, dockerized environments and git commits, and treats data like code. Cites surveys (90% see a reproducibility crisis, >50% fail to reproduce their own work) and concrete surprises (deep nets can reach 100% train accuracy on random labels; AlphaGo paper has ~6,300 citations but missing code and hyperparameters).
AI-written summary · about
chapters (13)
- 0:00Introduction to reproducible machine learning
- 1:35Reproducibility definitions
- 3:41Reproducibility crisis
- 8:18Why ML lacks reproducibility
- 11:30Introducing MLOps
- 13:37Data versioning in ML
- 16:59Kilt for data versioning
- 17:22MLflow experiment management
- 21:05Tracking MLflow experiments
- 22:33Deploying with MLflow Projects
- 24:07Storing models with MLflow
- 25:40Summary and tools
- 26:49Q&A on MLflow
related talks
29mReproducibility in ML Systems: A Netflix Original
30mReproducible Machine Learning at Scale
01h 16mReproducible machine learning and science with python
32mReproducibility and versioning of ML systems
33mBuilding Reproducible ML Processes with an Open Source Stack
38mHidden Gems in MLFlow that Improve Experiment Reproducibility
20mBetter Care and Feeding of Machine Learning Models
30mRealizing End to End Reproducible Machine Learning on Kubernetes