Kubernetizing Big Data and ML Workloads at Uber
Talkat KubeCon + CloudNativeCon North America 2019 · on Thursday, 2019-11-21· at 11:50· Room 15AB - San Diego Convention Center, Mezzanine Level·
summary
Uber plans to move Hive/Spark workloads off YARN to Kubernetes to save millions by co‑locating BigData/ML and microservices, and re‑architects Peloton as Peloton as Kubernetes plugins to regain YARN features. They implement hierarchical resource pools for reservations/limits/shares, build a remote shuffle service that matches local‑shuffle performance, add gang scheduling and GPU placement, and run thousands of jobs in production with on‑par latency while enabling dynamic partitioning and node migration for colocation.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:03Big Data and ML Use Cases at Uber
- 2:44Uber's Big Data Stack
- 7:50Peloton Overview
- 9:20Hierarchical Resource Pools
- 10:29Peloton Scheduler Architecture
- 11:55Elastic Resource Pool Example (cont.)
- 17:09Spark on Kubernetes Challenges
- 18:26How Does Spark Shuffle Service Work?
- 19:31Limitations of Spark Shuffle Service
- 24:01Remote Spark Shuffle Service Production Status
- 24:32Deep Learning Use Cases
- 24:52Distributed TensorFlow Challenges
- 25:22Gang Scheduling
- 27:21Colocating Batch and Stateless Workload
- 29:28Oversubscription
- 30:13Dynamic Partition Collocation Architecture
related talks
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
24mMigrating 2 million CPU cores to Kubernetes
32mMigrating 2 million CPU cores to Kubernetes
29mScaling Apache Spark on Kube to Apple Scale
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers
35mModern Big Data Pipelines over Kubernetes [I]
24mDynamic Big Data in the Cloud with Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes