End to End Collaborative Data Workflows
summary
Builds a Netflix‑style recommender from the MovieLens 20M CSVs using Google Cloud Storage, Dataprep (which compiles to an Apache Beam job pushed to Dataflow), BigQuery for exploration, Cloud Datalab + TensorFlow for prototyping, and Cloud ML Engine for hypertuned training and serving; a matrix‑factorization run takes about two hours on modest workers, the model is deployed as a versioned online endpoint and returns predictions via gcloud ml‑engine predict or an App Engine API. MovieLens 20M Dataprep -> Dataflow
AI-written summary · about
chapters (8)
- 0:00Introduction and overview
- 2:33Breaking down the challenge
- 4:48Transforming and understanding data
- 11:53Building machine learning models
- 15:52Deploying models at scale
- 20:08Live demo walkthrough
- 39:53Finalizing the workflow
- 42:44Next steps and resources
related talks
32mResurrecting a Recommendations Platform
57mHands-On with LLM-Powered Recommenders: Hybrid Architectures
30mHuman-Centric ML Infrastructure at Netflix
26mETL pipeline with GCP Dataflow
01h 21mUnlocking Personalization: A Deep Dive into Modern Recommendation Algorithms
30mDetect machine learning (ML) model drift in production
49mOrchestrating data and ML workflows with Apache Airflow
41mRecommendations for Building Machine Learning Software