100 machines do 100s work in 1s
summary
Claim: deploy a pool of 100 nodes from a shared image and run one consumer task per VM to process a large Service Bus queue in seconds. Uses C#/.NET Core, Azure Service Bus for messaging, Azure Batch for VM orchestration, Storage/Shared Image Gallery for binaries and images, and an Azure AD app plus the Batch management library to create 100 tasks automatically. Demo shows thousands of queued math jobs finish in seconds instead of ~25 minutes.
AI-written summary · about
chapters (8)
- 0:00Start
- 3:40Storage Account
- 3:57Batch Account
- 7:20Service Bus
- 14:59Visual Studio
- 24:12Startup Task
- 27:37Batch Controller
- 33:20Recap
related talks
56mDistributed Messaging patterns
52mDIY Async Message Pump: Lessons from the trenches
34mTen-year Journey to 10,000 Production Machines
11mRemote execution with 100 to 100,000 CPUs
43mAzure Messaging – Streams, Queues, and Pub Sub
16mBurst Computing: Quick, Sudden, Massively Parallel Processing on Serverless...
01h 00mScaling Batch Applications
14mScaling Large Production Clusters with Partitioned Synchronization