Efficient cluster resource management by using Cook and Mesos
summary
Two Sigma builds a scheduler on Mesos called Cook that converts user-declared task importance into quality scores and allocates memory/CPU with a matcher, lancer and rebalance loop; a simulation platform runs tens of thousands of runs daily to tune policies and shows median cluster utilization ~60% with a tail of ~10% low-utilization jobs while targeted approaches aim for 95–100% utilization and pay about 2–3% latency cost in interactive cases; system is open source and runs stably in production as Cook on Mesos.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:53What is Mesos and its features
- 4:20Mesos resource management
- 8:41Mesos at Two Sigma
- 12:39Fair resource allocation problems
- 17:31Cluster utilization optimization
- 22:52Formalizing resource fairness
- 26:25Mesos and Cook architecture
- 31:01Efficiency and utilization
- 32:23Open source and conclusion
- 33:04Q&A
related talks
34mSimulating Our Apache Mesos Framework for Fun and Profit and...
41mIdle Resources Oversubscription: Improving Cluster Utilization
38mMesomatic: The Cluster is a Library
27mKeynote Panel: Future of Cluster Management
29mGetting Started with Apache Mesos
39mPractical Container Scheduling: Juggling Optimizations, Guarantees, and Trade-Offs at Netflix
59mBuilding Distributed Systems with Apache Mesos
58mDistributed Scheduling with Apache Mesos in the Cloud