The Hidden Cost of ML Data Lifecycles in Kubernetes
summary
Personal churn‑prediction pipeline runs on Kubernetes with one PVC as shared datastore and three Kubernetes jobs (feature, training, inspect); over 11 days results quietly degrade because training reads an environment‑specified version while feature files accumulate (five filenames) and metadata is never verified, so the model trains on stale data. Rebuild enforces artifact versioning in filenames and JSON and makes version mismatch fail the pipeline, moving data storage outside Kubernetes.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:04Talk on ML data lifecycles
- 3:17Disappearing data version
- 8:19Inconsistent file naming broke pipeline
- 10:45Model output degradation over time
- 15:23Kubernetes manages compute not data
- 17:53Cost of ignoring data governance
- 20:45Four mistakes in ML pipeline
- 22:07Three rules for ML data discipline
- 24:13Key takeaways
related talks
33mCompliant Data Management and Machine Learning on Kubernetes
37mA Day in the Life of a Data Scientist. Conquer ML Lifecycle on Kubernetes
31mApplied MLOps to Maintain Model Freshness on Kubernetes
28mUnravelling Hidden Technical Debt in ML: A Pythonic Approach to Robust Systems
10mOptimizing Machine Learning (ML Ops) in Financial Services with Kubernetes and Cloud Technologies.
30mChallenges of machine learning: The new ML lifecycle
28mMachine Learning on Kubernetes at Shell: A Kubeflow Journey
21mMastering the DevOps Machine Learning Pipeline