From Jupyter to Spark: Stages of Machine Learning Productionization

on YouTube ↗17m

" summary

This presentation describes how Deezer moved machine-learning work from Jupyter notebooks into production on Apache Spark to provide real-time model predictions. It covers steps used for those migrations—including data architecture reviews and implementing models in Scala Spark—and illustrates the process with Deezer’s churn-prediction project and the collaboration between data scientists and data engineers.

AI-written summary · about

" chapters (13)

1. 0:00Intro
2. 1:11Deezer at a glance
3. 1:32Data Science for product and business
4. 3:54From ML exploration to deployment
5. 4:43What is churn at Deezer?
6. 5:24Identify business needs
7. 6:20Exploratory work
8. 8:18How can we go further?
9. 9:58Prerequisites for deployment
10. 11:31We need collaboration
11. 11:57Agreed on a solution for churn modeling
12. 13:36The long journey of code migration
13. 16:14Conclusion

" related talks

• 17mProductionizing Data Science WorkloadsAnand Iragavarapu · PyCon India 2020
• 39mMachine Learning with Scala on SparkJose Quesada · Scala Days Berlin 2016
• 40mThe data scientists are gone. how do I deploy their work to prod?Cédric Charlier · DevFest Lille 2019
• 51mSpotlight on Emerging Data Science Worskspaces and Spark as Development LeversCarlo Appugliese · ApacheCon North America 2017
• 43mDeploying Large Spark Models to production #bbuzzSubhojit Banerjee · Berlin Buzzwords 2018
• 42mHow mobile.de brings Data Science to Production...Florian Wilhelm, Markus Schüler · PyData Berlin 2018
• 23mRethinking scalable machine learning with Spark ecosystemAdi Polak · Scale By the Bay 2021
• 22mBringing Deep Learning into the Big Data AnalyticsYuhao Yang · Scale By the Bay 2017