What would you do with 10^6 cores? High Performance Computing in the AWS Cloud
summary
Western Digital runs 1,000,000 vCPUs (about half a million cores) on EC2 to do magnetic simulations using Docker on ECR and spot instances, cutting cost roughly 80% spot savings and dropping run time from three weeks to eight hours; another job hit 1.1 million vCPUs for topic classification. Core stack details: EFA low-latency MPI bypasses TCP/IP for scalable message passing, FSx Lustre maps S3 for parallel I/O, ParallelCluster (pip) automates cluster creation, and instance choices include P3dn, C5n, AMD, A1 ARM and upcoming Inferentia.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 0:30Frank Munz
- 2:12Brain Imaging
- 5:00High Performance Computing
- 6:15Cloud Computing
- 8:43HPC vs Cloud
- 10:17Compute Storage Networking
- 11:35Instance Types
- 13:52GPU Instance Types
- 14:23CPU Instance Types
- 16:09AMD Store
- 17:03ARM Instances
- 18:59Inferential
- 19:48Networking
- 23:01Lustre
- 23:42AWS FX Lustre
- 23:56AWS FX File System
- 24:32Parallel Cluster
- 26:05Testing
- 26:44Configuration
- 28:49Create Cluster
- 31:54Check Cluster Queue
- 35:04We choose to go to the moon
- 35:35Brain images
- 37:04The conclusion
- 37:27Contact details
- 37:59Output
related talks
34mHPC on AWS: Innovation without infrastructure constraints
43mWhat’s new in Amazon EC2
36mIntroduction to High Performance Computing on Google Cloud Platform
33mHPC on AWS: Innovation Without Infrastructure Constraints
42mHow Intel & AWS solve some of the world’s most urgent issues
17mOptimizing Cloud Compute Resources with Spare Cores
47mPowering global-scale predictive intelligence using HPC on
42mBuilding a green supercomputer in the cloud