Running Remote Shuffle Service to Solve a Well-Known Challenge for Apa...
Talkat KubeCon + CloudNativeCon + Open Source Summit China 2023 · on Wednesday, 2023-09-27· at 11:50· 2层 会议室 2 | 2F Room 2·
summary
Kubernetes Spark offloads shuffle data to an open remote shuffle service (Apache Calon / RSS) to decouple storage and compute, enabling true dynamic executor scaling. Tests show default shuffle-tracking (60s) wastes nodes, aggressive timeout (5s) causes fetch failures, while using Calon with executor idle timeout 10s yields responsive scaling and lower recompute. Two deployment modes exist (standalone or in-cluster) and benchmarking reports up to 38% cost savings with Apache Calon integration.
AI-written summary · about
chapters (6)
- 0:00Introduction and overview
- 3:02Spark on Kubernetes basics
- 4:34Challenges of dynamic allocation
- 13:42Apache Celeborn solution
- 19:27Deep dive into Celeborn
- 37:30Q&A session
related talks
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
26mOptimal Approaches for Real-Time Machine Learning with Apache Spark on Kubernetes.
29mScaling Apache Spark on Kube to Apple Scale
43mBeyond Shuffling: Scaling Apache Spark
31mManaging Multi-Cloud Apache Spark on Kubernetes
29mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
21mSpark on Kubernetes: The Elastic Story
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers