PopFetcher: Towards Accelerated Mixture-of-Experts Training Via Popularity...
summary
PopFetcher implements sliding-window popularity prediction to identify hot experts and does expert-wise prefetching of their parameters into local GPU memory during non-MoE computation, overlapping fetches with forward/backward work and prioritizing All-to-All streams to reduce contention. It formulates end-to-end latency, applies a pruning rule to fit GPU memory and link bandwidth, and synchronizes gradients for prefetched replicas. Implemented as a plugin for Megatron-like stacks and evaluated on two GPU clusters, it cuts training time by 15%–94.5% and outperforms DeepSpeed and other baselines.
AI-written summary · about
chapters (12)
- 0:00PopFetcher introduction
- 2:28MoE architecture challenges
- 5:14Expert parallelism bottlenecks
- 7:02Hybrid fusion prefetch
- 9:08Prefetching implementation challenges
- 10:33Dynamic expert popularity
- 12:42Expert pruning and latency
- 13:49Scheduling and resources
- 15:17Experimental setup
- 16:44Performance results
- 18:11Comparison with methods
- 19:12Conclusion and future
related talks
19mOffloading-Efficient Sparse AI Systems
12mSmartMoE: Efficiently Training Sparsely-Activated Models through Combiningand...
26mDeepSpeed: Efficient Training Scalability for Deep Learning
37mUnstructured Sparsity Meets Tensor Cores: Lessons from Sparse Attention and MoE
31mVeRL: Extreme Optimization Practices for Ultra-Large-Scale MoE Models
22mOptimizing Large MoE Inference on NVIDIA Blackwell: NVFP4, ADP, and DualPipe Strat...
36mAccelerating High-Performance Machine Learning at Scale i...
28mServing a Frontier Open Model with Ray & vLLM