Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
Talkat Data on Kubernetes Day North America 2023 · on Monday, 2023-11-06· at 16:25· W471ab, McCormick Place·
summary
A live demo builds an EKS cluster with Argo Workflows and Argo Events, an SQS event source, and the Spark operator to launch a DAG of Spark jobs; Carpenter auto-scales nodes and Apache Unicorn provides gang scheduling so drivers run on on‑demand instances and executors on spot instances. Networking guidance includes using a large secondary CIDR to reach ~120,000 pods, node‑local DNS and cluster‑proportional CoreDNS autoscaling, and NVMe local storage for heavy shuffles; the pipeline runs end‑to‑end in about two minutes.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:25Spark on Kubernetes
- 3:52Run Spark on K8s
- 8:22Operator vs submit
- 11:08Scaling challenges
- 12:10Networking and IP fixes
- 13:56CoreDNS scaling
- 15:01Storage for Spark
- 16:45Compute and cost
- 18:08Carpenter scheduling
- 18:52Unicorn scheduler
- 19:38Logging with Fluent Bit
- 20:00Argo Workflows
- 21:03Argo Events pipelines
- 22:48Demo setup
- 27:03Demo execution
related talks
29mDynamic Large Scale Spark on Kubernetes
36mAutomating Cloud-native Spark Jobs with Argo Workflows
14mLightning Talk: Running Cloud-native Spark Jobs with Argo Workflows
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
21mSpark on Kubernetes: The Elastic Story
30mBeyond Experimental: Spark on Kubernetes
27mBuilding a Data Science Platform with Argo Workflows
23mArgo at Scale: Navigating Complex Multi-Dimensional Deployments Across H...