Human-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
summary
Metaflow + Kubernetes integration lets data scientists write Python DAGs locally and transparently scale steps to Kubernetes or AWS Batch while keeping every run versioned and resumable; resource decorators specify GPU or memory per step, logs stream back to the laptop, and flows compile to orchestrators like Argo or Step Functions for production. Demo runs parallel jobs (five-way array example) and Intuit reports 5,000+ concurrent pods in production; supports S3/GCS/Azure blob stores and resuming from any arbitrary step successfully.
AI-written summary · about
chapters (9)
- 0:00Data science infrastructure challenges
- 3:49Local prototyping to production
- 10:50Metaflow model and Kubernetes
- 15:45Demo and local execution
- 17:56Scaling with Kubernetes and Argo
- 23:30Production deployment and scheduling
- 24:32Debugging production issues
- 28:16Metaflow integrations
- 30:26Q&A on GPU and on-prem
related talks
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
24mHuman Friendly, Production Ready Data Science Stack
44mHow Metaflow Became Netflix's Beloved Data Science Framework
27mMore Data Science, Less Engineering
01h 29mFull-stack Machine Learning and Generative AI for Data Scientists
36mEnterprise-grade Full Stack ML Platform
35mBuilding Robust Reactive ML Systems In A Multiplayer Setting
27mApplied Machine Learning: a Netflix production