Unlock the Future of Generative AI: TorchTitan's Latest...
summary
TorchTitan adds concrete systems: 5D parallelism (context-sharded sequence dimension plus expert parallel for Mixture of Experts), compiler-driven parallelism with simple FSDP and auto-parallel, and fault-tolerant / semi-sync training via torch FFT. Supported models extend beyond Llama 3 to DeepSeek v3 (MoE), GPOSS, Flux diffusion and SikLip VLMs. DeepSeek v3 hits MFU ≈ 7.5 with torch.compile (higher if all‑to‑all is enabled). Hugging Face loading/publishing, cloud and RL integrations are production-ready.
AI-written summary · about
chapters (8)
- 0:00TorchTitan introduction and 2024
- 1:29Community features and 2025
- 3:14DeepSeek V3 benchmarks and improvements
- 4:57TorchTitan modular framework
- 6:24Hugging Face and RL integration
- 8:33Adoption by labs and clouds
- 9:55Academic impact and research
- 10:37Next steps: efficiency and RL
related talks
25mInto Generative AI with PyTorch Lightning 2.0
23mTorchTPU: Expanding TPU Programmabil...
47mGet to Know PyTorch - A Cutting Edge Open Source AI Framework for Deep Learning
23mHow Pytorch Powers Training Inference
54mThe Future of AI: Building the Most Impactful Technology Together
23mBringing ExecuTorch To the Next Frontiers of Edge AI
11mLightning Talk: Trinity Large - Torchtitan on 2000+ B300s
09mTogether Goes Brrr: Threading Research & Production with Torch Compile