Spark in K8s for dozens of DS teams

on YouTube ↗52m

" summary

Describes how Spark workloads were implemented on a recommendation-system platform, detailing problems encountered with Spark and the reasons for the chosen solution. Explains how Spark is used in a multi-tenant architecture and discusses issues of running Spark on Kubernetes (Spark in K8s).

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 4:40Why Spark in K8s
3. 6:26Spark in K8s risks
4. 11:20Data locality loss
5. 15:17Intermediate data issues
6. 20:20Scaling setup
7. 24:39Spark operator benefits
8. 28:58Spark Connect and dynamic location
9. 33:28Multi-tenancy and isolation
10. 37:47Q&A

" related talks

• 29mDynamic Large Scale Spark on KubernetesOvidiu Valeanu · KCD Romania 2024
• 39mSpark on Kubernetes: Best Practice and PerformanceJunjie Chen, Jerry Shao · KubeCon + CloudNativeCon China 2018
• 21mSpark on Kubernetes: The Elastic StoryBowen Li, Huichao Zhao · KubeCon + CloudNativeCon Europe 2022
• 22mSPOK - Managing ML/Big Data Spark Workloads at scale on KubernetesNagaraj Janardhana, Mikhail Arov · OpML '20
• 26mBuilding Apache Spark Clusters on K8s for AI WorkloadsAnant Kumar · Conf42 Cloud Native 2025
• 30mBeyond Experimental: Spark on KubernetesWeiwei Yang · Kubernetes Batch + HPC Day North America 2022
• 41mRunning Apache Spark on K8s: From AWS EMR to K8sRamiro Alvarez Fernandez, Álvaro Panizo, Daniel Hernández Alfageme · Berlin Buzzwords 2022
• 40mApache Spark on Kubernetes: A Technical Deep DiveYinan Li · KubeCon + CloudNativeCon China 2018