Simplification of Data Pipelines by Containerization
summary
Claim: containerizing end-to-end ML pipelines makes iteration fast and operationally simple. Demo uses Spark 2.3 on Kubernetes, automated Docker builds via Cloud Build trigger from a Git tag, images in Container Registry, and a running 3-node n1-standard-4 cluster in Frankfurt with proper RBAC so the Spark driver can spawn three executor pods. Pipeline trains linear regression and random forest on the Kaggle Smart Meter London dataset; build takes ~3 minutes and the random forest produces a lower RMSE, so the flow works.
AI-written summary · about
chapters (15)
- 0:00Introduction to talk
- 1:06Data science overview
- 6:31Spark for big data
- 8:59Kubernetes as cluster manager
- 12:13Demo setup introduction
- 12:37Smart meter prediction goal
- 15:31Demo technologies explained
- 17:39GitHub and container registry
- 23:20Kubernetes cluster setup
- 26:23Submitting Spark job
- 28:41Model evaluation results
- 30:51Random forest regression test
- 32:02Iterative model improvement
- 32:46Company introduction
- 33:49Q&A and closing
related talks
35mOrganizing data engineering around Kubeflow
30mDeploy end to end ML pipeline using Apache spark ...
36mDemocratization of ML Pipelines and Bringing ML Workflows to Heterogeneous...
18mAutomate the boring ML stuff with pipelines
24mTaming Data Skew in Production ML Pipelines
23mRethinking scalable machine learning with Spark ecosystem
43mData Pipelines à la Mode
34mKick-Ass Tools and Processes for Scalable Data Science