Automating Cloud-native Spark Jobs with Argo Workflows
summary
Shows Argo Workflows + Kubernetes running containerized Apache Spark jobs so a single cluster hosts multiple Spark versions and avoids global installs. Demo submits a SparkApplication via the Spark Operator, pulls prebuilt images, runs a Kaggle bike‑share job that uses two executor pods and prints bike‑type counts and ride lengths. Workflow YAML and workflow templates with parameters simplify reuse, Argo monitors job state, and features like Kubernetes operator autoscale to zero and node‑pool GPU affinity are shown as working options.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:30BACKGROUND: APACHE SPARK
- 4:51BACKGROUND: KUBERNETES
- 8:15BACKGROUND: ARGO WORKFLOWS
- 22:16NEXT STEPS
- 27:30APPENDIX: ARGO WORKFLOWS VS. APACHE AIRFLOW
related talks
14mLightning Talk: Running Cloud-native Spark Jobs with Argo Workflows
31mDynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
29mDynamic Large Scale Spark on Kubernetes
52mCloud-Native Kubernetes Workflows on AKS with Argo
52mCloud-Native Kubernetes Workflows on AKS with Argo
39mBuild and Deploy Your Application with Argo on Kubernetes
27mLeveraging Argo Workflows To Move, Migrate and Manage All of Your Workflows
27mBuilding a Data Science Platform with Argo Workflows