Realizing End to End Reproducible Machine Learning on Kubernetes
Talkat KubeCon + CloudNativeCon North America 2019 · on Wednesday, 2019-11-20· at 16:25· Room 16AB - San Diego Convention Center, Mezzanine Level·
summary
Nearmap processes petabyte-scale aerial imagery and demonstrates an end-to-end reproducible ML pipeline on Kubernetes that actually runs at scale: 2,000 GPUs for two weeks to label a million+ km at 5cm/pixel using preemptible instances. The stack wires Pachyderm + Kubeflow + Seldon for data provenance, tuning and serving; pipelines inject version IDs so models and calibration weights are retrievable. Reproducibility relies on global seed & determinism flags, pinned libraries and accepted tradeoffs between full determinism and speed.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:30Agenda
- 4:30Why Reproducible Matters
- 6:30Challenges in Machine Learning
- 11:00The Middle Ground
- 12:30Reproducible ML Code
- 15:00Reproducible ML Results
- 16:00Version Control
- 18:00Workflow
- 20:00Release
- 21:00Tools
- 23:00Cubeflow
- 27:00Replicability
related talks
33mCompliant Data Management and Machine Learning on Kubernetes
51mMachine Learning using Kubeflow and Kubernetes
39mEnd to End Machine learning pipelines for Python driven organizations
30mReproducible Machine Learning at Scale
28mMachine Learning on Kubernetes at Shell: A Kubeflow Journey
37mA Day in the Life of a Data Scientist. Conquer ML Lifecycle on Kubernetes
49mKubernetes and MLOps for Scalable and Reproducible Generative AI
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines