Lightning Talk: Scaling Recommendation Systems To 2K GPUs and Beyond
summary
Meta describes a production system that implements 2D sparse parallelism to shard massive embedding tables across hierarchical model plus data parallel groups, moving from ~500 GPU runs to stable 8k GPUs (demonstrated at 4k). The implementation uses PyTorch primitives, fused backward+optimizer kernels and an embedding table all-reduce (syncing weights with scaled learning rates) and reports 10–20% QPS gains, lower lookup latency, and ≈20 GB peak-memory savings.
AI-written summary · about
related talks
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
25mScaling Inference of O(10K)-length Sequence Recommendation Models Using...
13mFrom Silicon to Scale: Training at Scale
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
14mLightning Talk: Large-Scale Distributed Training with Dynamo and...
24mTraining large scale models using PyTorch