More Data Science, Less Engineering

on YouTube ↗27m

" summary

Netflix uses an open source Metaflow framework to let data scientists write idiomatic Python DAGs with decorators that declare steps, automatic state persistence and versioned snapshots, and isolated executions with unique IDs. It provisions remote machines per-step via an @resources spec, parallelizes branches and a for each primitive for grid searches, captures conda environments for reproducibility, and exports workflows to production schedulers (Step Functions, Airflow). The platform is production-used across content, infra, subtitles and fraud problems.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 0:35Machine learning at Netflix
3. 3:24Data science stack challenges
4. 9:19Introducing Metaflow
5. 13:35Scaling and resource management
6. 18:47Dependencies and reproducibility
7. 21:52Deployment and production
8. 25:28Conclusion

" related talks

• 44mHow Metaflow Became Netflix's Beloved Data Science FrameworkJulie Amundson · YOW! Melbourne 2022
• 27mApplied Machine Learning: a Netflix productionJulie Pitt · Scale By The Bay 2018
• 32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & KubernetesSavin Goyal, Saravanan · KubeCon + CloudNativeCon North America 2022
• 30mHuman-Centric ML Infrastructure at NetflixVille Tuulos, Savin Goyal · Scale By The Bay 2019
• 36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & KubernetesGoyal · PyData NYC 2022
• 03mInterviewJulie Pitt · Scale By The Bay 2018
• 04mInterviewShiva Chaitanya · Scale By the Bay 2017
• 29mDagster: a Framework for Data Processing ApplicationsNick Shrock · Scale By The Bay 2019