Panel Discussion: Training and Inference at Planet Scale
summary
Panel covers concrete engineering moves: using five- or six-dimensional parallelism and mixture-of-experts sharding, splitting prefill vs decode into separate pools, and experimenting with attention-FFN disaggregation to map parts of inference to different hardware. Tools discussed include dtensor/torchcompile, Ray for RL orchestration, vLLM and Fireworks optimizations, and limits of NCCL for dynamic links. Claims: deterministic inference mode is implemented, hundreds-of-CUDA-graphs per VM cause warmup/memory pain, and teams spend 20+ days just to match reference outputs.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:24Distributed training trends
- 3:50Fine-grained model parallelism
- 5:14Inference for long contexts
- 7:00Data curation in training
- 7:43Reinforcement learning with Ray
- 9:51Debugging large systems
- 10:54Deterministic inference
- 11:55RL training and inference
- 13:20Model to system complexity
- 14:42Debugging at scale
- 15:25Fault tolerance in GPUs
- 18:14PyTorch feature requests
- 22:47Eager vs compile-first
- 25:41Synchronous vs asynchronous training
- 28:50Batching and latency trade-offs
- 31:21NCCL and sharding limits
related talks
33mLive Q&A Session 2
11mEngineering Scalable ML Infrastructure for Modern Apps
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
01h 03mScaling up machine learning without tears (and what do programming languages ha...
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
26mAmazingly Fast and Incredibly Scalable Inference...
24mTraining large scale models using PyTorch