Compute anything with Metaflow
summary
Metaflow claims to let Python workflows pick the actual hardware and run reliably: cloud or on‑prem, packaged dependencies, Kubernetes scheduling and GPU boxes, plus a month‑old distributed training on large GPU clusters feature that Netflix‑originated Metaflow already uses in production. Examples include asking for 64 GB kubernetes memory or an A100 GPU for a single step, and performance anecdotes where vertical boxes beat heavy Spark jobs; historical scale cited as a 768× increase in largest cloud instance memory.
AI-written summary · about
chapters (6)
- 0:00Metaflow and compute history
- 7:54Horizontal scalability limits
- 13:50Cloud and heterogeneous compute
- 18:46Python in compute frameworks
- 23:39Compute paradigm selection
- 27:51Metaflow features and future
related talks
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
35mResource Monitoring and Optimization with Metaflow
30mNavigating Cloud Expenses in Data & AI: Strategies for Scientists and Engineers
36mEnterprise-grade Full Stack ML Platform
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
19mHow Intuit Streamlined AI/ML Inference Workflows on K8s
31mMLflow: Platform for Complete Machine Learning Lifecycle
21mCapacity Planning for AI Training Hardware