Our Journey With TorchTitan
summary
IBM Research moves from throughput benchmarks to production using TorchTitan and trains an in-house 70B model matching LLaMA 3 while consuming roughly one‑third the GPU hours. FP8 adoption yields about 1–1.5x throughput and halved token budget via FP8 FSDP all‑gathers. Production additions include async offload checkpointing, Weights & Biases and Hugging Face adapters, gradient accumulation, rescalable tokenizer‑aware dataloaders, and an integrated fast MoE kernel enabling expert‑parallel training; GPU‑initiated all‑to‑all support is planned.
AI-written summary · about
chapters (16)
- 0:00TorchTitan journey intro
- 1:26What is TorchTitan
- 2:50Early eval and collaboration
- 4:37Activation checkpointing
- 5:44Llama 3 70B training start
- 7:49Production readiness
- 9:54Data loading scalability
- 11:41Full pre-training results
- 13:27Model architecture trends
- 14:53Mixture of experts support
- 17:43Custom model integration
- 18:48Testing Spire accelerator
- 20:59TorchTitan as documentation resource
- 23:04Dev advice and contributions
- 24:27Target users and use cases
- 27:27Q&A
related talks
10mKeynote: How to Leverage PyTorch to Scale AI Training and Inferencing
11mLightning Talk: Trinity Large - Torchtitan on 2000+ B300s
09mTogether Goes Brrr: Threading Research & Production with Torch Compile
01h 40mFrom model weights to API endpoint with TensorRT LLM
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
25mMoving Inference to Triton Servers
13mFrom Silicon to Scale: Training at Scale
01h 18mOptimizing LLM Training and Inference Performance on GPUs