Beyond Experimental: Spark on Kubernetes
summary
Builds a cloud‑native Spark platform that hides multiple Kubernetes clusters behind a batch processing gateway which accepts job specs and maps virtual queues onto one or more physical clusters; each cluster runs a Spark K8s operator and a per‑cluster Apache YuniKorn scheduler that implements gang scheduling, hierarchical queues and bin‑packing. Jobs run on a single cluster, Unicorn shows ~2x scheduler throughput versus kube‑scheduler, and operational guidance is to avoid single clusters beyond ~1,000 nodes, use cluster autoscaling, Fluentd logging, and explore remote shuffle.
AI-written summary · about
chapters (11)
- 0:00Start
- 1:26Apache Spark meets Cloud Native
- 2:51Architecture Evolution: Giant Compute Cluster
- 4:58Architecture Evolution: Cloud - Dynamic Compute Pools
- 8:36The Recipe
- 9:33A typical workflow
- 11:28Batch processing gateway
- 12:39Delight users = Hide complexity
- 14:04Virtual resource queues
- 16:57Challenges
- 19:18Scale Tips
related talks
21mSpark on Kubernetes: The Elastic Story
36mNext Level Spark on Kubernetes with Apache YuniKorn (Incubating)
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
29mDynamic Large Scale Spark on Kubernetes
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers
40mApache Spark on Kubernetes: A Technical Deep Dive
31mManaging Multi-Cloud Apache Spark on Kubernetes
37mSpark and Flink Running Scalable in Kubernetes #bbuzz