Designing High-Performance Scalable Middleware for HPC, AI, & Data Science in Exascale Systems & ...
summary
mvapich2 and mvapich2-gdr present concrete middleware wins: sub-200ns intra-node and ~1.9µs inter-node MPI latency, 41.9 GB/s intra-node bandwidth, and job startup of 229,000 processes in 31s. Sharp in-network offload gives 5–9x faster collectives; DC transport and cooperative rendezvous yield ~19% Graph500 gain; optimized async progress speeds p3dfft/HPL by ~33%/29%; XPmem zero-copy gives up to 4x reduce. GPU device-to-device cross-node hits 1.85µs (≈10x over naive), on-the-fly compression cuts comms and runtime for seismic and ImageNet workloads (≈500k imgs/s), and a pathology pipeline drops 32h to 27min.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:47Supporting Programming Models for Multi-Petaflop and Exaflop Systems: Challenges
- 2:15Designing (MPX) Programming Models at Exascale
- 3:30Overview of the MVAPICH2 Project
- 5:11MVAPICH2 Release Timeline and Downloads
- 5:26Architecture of MVAPICH2 Software Family for HPC, DL/ML, and Data Science
- 7:03Highlights of MVAPICH2 2.3.6-GA Release
- 8:17Startup Performance on TACC Frontera
- 9:06Performance of Collectives with SHARP on TACC Frontera
- 10:25Performance Engineering Applications using MVAPICH2 and TAU
- 12:04Overview of Some of the MVAPICH2-X Features
- 12:31Impact of DC Transport Protocol on Neuron
- 14:23Cooperative Rendezvous Protocols
- 15:30Benefits of the New Asynchronous Progress Design: Broadwell + InfiniBand
- 16:33Shared Address Space (XPMEM)-based Collectives Design
- 19:13MVAPICH2-GDR 2.3.6
- 20:20Highlights of some MVAPICH2-GDR Features for HPC, DL, ML and Data Science
- 23:23MVAPICH2-GDR with CUDA-aware MPI Support
- 26:38Performance with On-the-fly Compression Support in MVAPICH2-GDR
- 28:52Collectives Performance on DGX2-A100 - Small Message
- 29:59MVAPICH2 (MPI)-driven Infrastructure for ML/DL Training
- 31:17Distributed TensorFlow on ORNL Summit 1,536 GPUS
- 32:32Distributed TensorFlow on TACC Frontera (2048 CPU nodes)
- 33:10PyTorch, Horovod and DeepSpeed at Scale: Training ResNet-50 on 256 V100 GPUs
- 37:42Dask Architecture
- 38:13Benchmark #1: Sum of cupy Array and its Transpose (12)
- 39:18Benchmark #2: cuDF Merge (TACC Frontera GPU Subsystem)
- 39:54MVAPICH2-GDR Upcoming Features for HPC and DL
- 45:07Funding Acknowledgments
related talks
45mHigh-Performance and Scalable Middleware for HPC and Deep...
25mHigh-Performance MPI and Deep Learning with Introspection...
31mHigh-Performance MPI and Deep Learning on OpenPOWER Platform
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
39mAccelerating Data Science with RAPIDS
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale
18mCo-Designing Communication for AI Accelerators
36mAn almost perfect match: cloud native and scientific computing