Human-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
summary
Metaflow claims to already power hundreds of ML projects across domains and provides concrete features: automatic execution snapshots to S3 that record code, data and artifacts; a Kubernetes decorator runs steps remotely so heavy experiments lift off local laptops; integration points with Airflow/Argo and cloud providers (AWS, GCP) for production orchestration; dependency snapshotting and per-run IDs for reproducibility. Demos show workflows scaled, retried, and debugged end to end, not just slides about concepts.
AI-written summary · about
chapters (9)
- 0:00Metaflow & Kubernetes intro
- 3:52Scaling data science workflows
- 9:02Ideal data science stack
- 13:27Metaflow for production workflows
- 18:30Orchestrating with Metaflow & Kubernetes
- 21:36Live Metaflow workflow demo
- 27:45Kubernetes cluster workflows
- 30:23Debugging failed workflows
- 32:37Q&A
related talks
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
44mHow Metaflow Became Netflix's Beloved Data Science Framework
36mEnterprise-grade Full Stack ML Platform
28mCompute anything with Metaflow
01h 29mFull-stack Machine Learning and Generative AI for Data Scientists
27mApplied Machine Learning: a Netflix production
35mBuilding Robust Reactive ML Systems In A Multiplayer Setting
27mMore Data Science, Less Engineering