Applying DevOps practices in Data and ML Engineering
summary
Argues that data and ML need the same DevOps rigor as apps and demonstrates an open source stack to get there: Versatile Data Kit (SDK + control service) that builds, containers, tests and deploys data jobs, collects metrics and enforces data API contracts and SLOs. Shows plugins that intercept queries (example: reject queries >1000 expressions or >1MB) and integrations with Snowflake/Redshift/MySQL/Spark and Kubernetes. Claims this automates ingestion, transforms, validation and monitoring so data APIs become testable, versioned and operable.
AI-written summary · about
chapters (19)
- 0:00intro
- 0:27preamble
- 0:40about 'moneyball'
- 2:19agenda
- 2:39applications
- 3:41data applications
- 4:42data journey
- 6:40api
- 6:59api components
- 7:47api for data example
- 10:39api for data
- 12:25slos and slas for data
- 13:56devops cycle for data
- 15:48versatile data kit
- 17:03versatile data kit sdk
- 18:20what are we going to do?
- 20:50versatile data kit control service
- 23:34dataops for data
- 25:14thank you
related talks
56mDevOps for Machine Learning
30mDevOps For Data Teams
29mDevOps for Data Science? - automate the boring stuff and leverage the OSS ecosystem
44mDevOps for Machine Learning - why is it different?
51mFrom DevOps for data to DataOps
59mDevOps for Machine Learning Projects
59mDevOps for Data Science
01mMLOps: Applying DevOps to ML