Kick-Ass Tools and Processes for Scalable Data Science
summary
Shows a working pipeline that runs Spark on Kubernetes (Spark 2.3/2.4) deployed to AWS EKS, using CloudFormation to create auto-scaling EC2 nodes and ECR-hosted Docker images. Processes Wikipedia clickstream in a demo job submitted with spark-submit, fires up driver and executor pods, runs parallel data-frame queries, and writes results in parallel to S3. Cluster scales up for the job and tears down resources afterwards; the demo completes successfully.
AI-written summary · about
chapters (7)
- 0:00Start
- 1:49The master plan
- 6:02Scaling data science
- 9:02Strategy
- 9:42Tech stack
- 11:24Apache Spark
- 14:45Demo
related talks
29mDynamic Large Scale Spark on Kubernetes
31mSpark challenges
26mRe-architecting the Data Platform with Spark on Kubernetes
31mDynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
42mBuilding Robust Streaming Data Pipelines with Apache Spark
41mBuilding a modern data platform on Amazon EKS
36mAutomating Cloud-native Spark Jobs with Argo Workflows
33mSpark Operator - Feature Engineering with Spark on Kubeflow