DevOps in Data Science: What Works and What Doesn’t
summary
An engineering team builds a Kubeflow-based CI/CD pipeline using Kaniko + Kubeflow Pipelines to let data scientists self-build containers and uses ArgoCD for delivery, but the CI process proves overly complex and data scientists push back. The container-build + pipeline approach creates cognitive debt and fragile versioning; the team abandons it in favor of KServe lightweight serving, which reduces required Kubernetes knowledge and works better for production model deployment, while organizational processes and simple, pluggable APIs remain critical.
AI-written summary · about
chapters (9)
- 0:00DevOps in Data Science intro
- 2:29MLOps vs DevOps misalignment
- 5:16Data scientists’ needs beyond models
- 9:04Why the First Platform Failed
- 10:27DevOps helping data science
- 13:01Building services without reinventing
- 17:16Simplifying complexity with APIs
- 19:53DevOps in Data Science next steps
- 21:24Q&A
related talks
30mDevOps For Data Teams
59mDevOps for Data Science
29mDevOps for Data Science? - automate the boring stuff and leverage the OSS ecosystem
05mCI/CD Pipelines and Machine Learning
28mDevOps in a Data Warehouse: Inside Out
26mApplying DevOps practices in Data and ML Engineering
36mDataOps and MLOps: NLP Development and Deployment in Kubernetes made easy
35mOrganizing data engineering around Kubeflow