Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and
Talkat 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI '21) · 2021-07-14..07-16·
summary
Dorylus is a distributed system for training graph neural networks that combines CPU servers with serverless AWS Lambda threads to scale to billion-edge graphs while reducing cost. It separates computation into graph-parallel and tensor-parallel stages to build a bounded-asynchronous pipeline that overlaps work and hides Lambda network latency. The paper reports that adding Lambdas to CPU servers improves performance-per-dollar over CPU-only and outperforms GPU servers and existing sampling-based systems on large, sparse graphs.
AI-written summary · about
chapters (27)
- 0:00Intro
- 0:35Machine Learning
- 0:48Graph Neural Networks
- 1:24Stages of a Graph Neural Network
- 3:21GPUs Are Not a Good Fit for Graph Operations
- 4:46Combining CPUs and GPUs is Cost-Ineffective
- 5:07Using Many CPU Servers Can Still Be Expensive
- 5:32Key Insight: Serverless Fits Our Goals
- 6:16Serverless Achieves Low-Cost, Scalable Efficiency
- 6:26Challenges with Using Serverless
- 6:47Challenge 1: Limited Resources
- 6:57Solution: Computation Separation
- 7:18Dorylus Architecture
- 8:27Flow of Decomposed Tasks
- 8:46Challenge 2: Limited Network
- 9:00Solution: Create Pipeline of Decomposed Tasks
- 9:17Data Chunks Moving Through Layer of Pipeline
- 10:00Synchronize after Scatter Hinders Pipeline
- 10:12Two Sync Points Makes Asynchrony Difficult
- 10:34Minimizing Effects of Asynchrony on Convergence
- 11:12Serverless Optimizations
- 11:39Data Graphs
- 12:01We Evaluated Several Aspects of Dorylus
- 12:48High Value on Large-Sparse Graphs
- 13:29Dorylus Outperforms Existing Systems
- 13:58Dorylus Scales Full Graph Training
- 14:21Conclusion: Dorylus Provides Value
related talks
41mScaleable Distributed Deep Learning
27mHow to Eliminate the I/O Bottleneck and Continuously Feed the GPU While Training in the...
20mLegion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale...
34mDemocratizing Machine Learning on Kubernetes [I]
26mDeep Graph Library: Deep Graph learning at scale
18mGPU-Disaggregated Serving for Deep Learning Recommendation Models at Scale
34mBuilding scalable end-to-end deep learning pipelines in the cloud
19mNeuGraph: Parallel Deep Neural Network Computation on Large Graphs