Optimizing PyTorch on CPU-GPU Coherent Platforms
summary
Covers practical steps for GB200 and coherent platforms: avoid needless CPU→GPU duplication by using a custom transfer tensor class that solves dispatch, backing, and autograd; prefer pinned vs managed memory depending on streaming vs L2-cache needs; shows a real Llama 2 70B offload result with up to 60–67% throughput gain per GPU using multi‑stream offload (Transformer Engine) and explains the math: naive offload bandwidth needs (≈1,500 GB/s) drop with microbatching (≈750 GB/s) so partial offload can still improve throughput.
AI-written summary · about
chapters (8)
- 0:00CPU-GPU coherent systems
- 2:06CPU-GPU data transfers in PyTorch
- 7:01Unified memory on coherent platforms
- 10:32Optimizing Grace Hopper and Blackwell
- 13:20Memory bandwidth ratios
- 15:04CPU offloading for performance
- 23:48When to use offloading
- 26:57Q&A
related talks
24mInside MoE Optimization: A Profiler-Guided Tour of torch.compile and vLLM
08mHacks to Make LLM Training Faster
22mGenAI Training In Production: Software, Hardware & Network Considerations
18mHigh Performance LLM Inference in Pure Python with PyTorch Custom Ops
17mEnable Generative AI Everywhere with Ubiquitous Hardware and Open Software
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
22mPerformance Optimizations at 100K+ Scale