100 machines do 100s work in 1s
summary
Claim: 100 small VMs process hundreds of CPU‑bound messages in seconds by scaling out the same image and binaries. Implementation uses C#/.NET Core, Azure Batch to provision a 100‑node pool from a shared image, Service Bus queues for to‑do/results, startup tasks to pull app binaries, and a controller that creates 100 tasks. The job runs, the to‑do queue empties in minutes, logs land in each VM stdout, and the approach proves practical.
AI-written summary · about
chapters (7)
- 0:00Introduction to scaling
- 5:45Creating Azure resources
- 13:31Setting up service bus
- 18:37Preparing Linux VM
- 22:16Configuring Azure Batch
- 27:03Scaling with 100 tasks
- 32:17Results and conclusion
related talks
34mTen-year Journey to 10,000 Production Machines
11mRemote execution with 100 to 100,000 CPUs
56mDistributed Messaging patterns
16mBurst Computing: Quick, Sudden, Massively Parallel Processing on Serverless...
43mScalability Intro – Shyam Jeedigunta, Google & Bob Wise, AWS (Any Skill Level)
43mSystems @Scale Summer Q&A Week 2
46mWhat do I do with 1000 cores
42mThe Soul of Erlang and Elixir