How to Build a Cluster to Run Hundreds of Thousands of High-CPU/High-MEM Jobs
summary
A cloud CI service that runs customers' tests needed an architecture to quickly scale a cluster up and down for hundreds of thousands of high-CPU/high-memory test jobs. The talk describes using automated instance procurement (they acquired 37,218 Amazon instances) to process 189,488 test tasks and the operational challenges of unpredictable parallel load and rapid capacity elasticity.
AI-written summary · about
chapters (6)
- 0:00Start
- 6:46Ways to Solve Problems
- 9:18Does It Suit Us?
- 12:56Evolution of Vexor Architecture
- 14:51How Does It All Work?
- 21:12Optimization
related talks
13mOptimizing Workload Management in a Multi-Cluster Environment: A Case Study
36mMigrating Applications to the Cloud in a Cost Effective Manner
27mFrom Supercomputing to Serving: A Case Study Delivering Cloud Native Foundation Models
21mKeynote: CERN Experiences - Ricardo Rocha & Clenimar Filemon
47mCloud Native and Sustainable, Reproducible Scientific Computing
22mBuilding a Flexible, Scalable Infrastructure to Test Dolby.IO at Scale
01h 08mPerformance testing in the cloud
39mWhat would you do with 10^6 cores? High Performance Computing in the AWS Cloud