Morcor - Co-Location of Mixed Workloads at Uber
summary
Uber co-locates stateless microservices and preemptible batch jobs by dynamic partitioning and host overcommitment, running containerized YARN NodeManagers on Peloton-managed hosts (yarn-on-peloton). Deployment hits ~90% of fleet, frees hundreds of thousands of cores, and runs tens of thousands of batch jobs daily. The system autosizes partitions from live utilization, uses load-aware placement and proactive draining to avoid hotspots, and reclaim hosts by ordered cleanup and coordinated NodeManager/Docker teardown.
AI-written summary · about
chapters (8)
- 0:00Co-locating mixed workloads
- 3:34Stateless and batch workloads
- 7:26Same-cluster co-location at Uber
- 13:27Resource contention and resizing
- 14:52Peloton and YARN convergence
- 17:23YARN-on-Peloton integration
- 19:26Cost-saving co-location benefits
- 20:10Lessons learned and future
related talks
41mYARN Resource Management and Dynamic Max
27mKeynote Panel: Future of Cluster Management
20mAdvancing HPC The Hybrid Scheduler Approach at the D.E.Shaw Group
39mPractical Container Scheduling: Juggling Optimizations, Guarantees, and Trade-Offs at Netflix
40mNomad: Scheduling Applications at Scale
12mHow the Data Center Shapes the Workload
39mPanel: Cracking the Data Locality Puzzle
16mTowards Optimal Rack-scale μs-level CPU Scheduling through In-Network Workload...