Lightning Talk: Running Cloud-native Spark Jobs with Argo Workflows
summary
Demonstrates running Apache Spark on Kubernetes using Argo Workflows and the Spark Operator, submitting jobs in Cluster mode with Docker images, a Scala main class and jar path. Shows a DAG that runs two Spark jobs in parallel, each driver spawning two executor pods per driver, and captures logs via kubectl. Explains workflow template parameters for easy parameterization, mixing Spark versions (2.x and 3.1), node pool selection, autoscaling, and integration with Kafka/Argo Events for streaming to ML training.
AI-written summary · about
chapters (7)
- 0:00Introduction to the topic
- 1:09Spark on Kubernetes benefits
- 3:24Why Kubernetes for data
- 4:18Argo workflow orchestration
- 7:03Practical Spark demo
- 10:37Parameterizing workflows
- 11:54Advanced architecture and wrap
related talks
36mAutomating Cloud-native Spark Jobs with Argo Workflows
31mDynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
29mDynamic Large Scale Spark on Kubernetes
52mCloud-Native Kubernetes Workflows on AKS with Argo
52mCloud-Native Kubernetes Workflows on AKS with Argo
44mArgoCD + Spark + Kubernetes = a Managed Analytics Platform
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
22mCross Cluster Execution of Argo Workflows