Human-Centric ML Infrastructure at Netflix
summary
Metaflow packages each run’s Python code, Conda environment and immutable data snapshots so executions are reproducible across local laptop or cloud; it records every execution and builds a conda-style lockfile automatically. Runtime provisioning picks right-size EC2 instances (Netflix notes up to 768 GB RAM on r5 instances) and can fan-out identical jobs for hyperparameter search. Deployed at scale for recommendation, content and CDN problems across ~250 million members, it increases data scientist productivity by removing infra friction.
AI-written summary · about
chapters (12)
- 0:00Introduction to Netflix ML
- 1:30The problem of reproducibility
- 3:43Defining reproducibility
- 6:11Addressing versioning challenges
- 9:24Managing dependency hell
- 13:00Metaflow solutions
- 14:14Vertical vs horizontal scaling
- 16:04Horizontal scaling overheads
- 18:34Usability in data science
- 21:19AWS instance evolution
- 24:30Scalability in practice
- 26:57Metaflow scalability features
related talks
27mApplied Machine Learning: a Netflix production
44mHow Metaflow Became Netflix's Beloved Data Science Framework
27mMore Data Science, Less Engineering
29mEvolving the ML Platform Organisation at Netflix: A Case Study
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
04mInterview
29mReproducibility in ML Systems: A Netflix Original
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes