Deep learning on AWS with NVIDIA: From training to deployment
summary
Walks through an end-to-end stack: use TAO Toolkit to fine-tune and prune vision and speech models for up to 8x model size reduction and quantize to INT8, use NeMo Megatron recipes and a hyperparameter tool to train large LMs (175B takes ~43 days on 800 A100s, H100 + transformer engine claims ≈6x faster), and deploy with Triton Inference Server plus TensorRT optimizations (up to 36x CV, ~600x speech) and SageMaker integration.
AI-written summary · about
chapters (25)
- 0:00Intro
- 2:07GPU power from the cloud to the edge
- 7:24NVIDIA H100 supercharges LLMs
- 8:12Transformer engine
- 14:25The NVIDIA TAO stack
- 22:13Training and deploying LLMs is not for the faint of heart
- 24:15NeMo Megatron - Features
- 25:16Pipeline and Tensor parallelism for training
- 25:54NeMo Megatron - Optimization techniques
- 27:36Solving pain points across the stack
- 30:37Challenges in Al inference
- 32:03Inference is complex
- 34:18Layer and Tensor fusion
- 35:24Kernel auto-tuning
- 36:06Dynamic Tensor memory
- 37:34Multi-stream concurrent execution
- 39:42NVIDIA Triton Inference Server
- 41:55Triton's architecture
- 44:08Concurrent model execution
- 44:51Dynamic batching scheduler
- 45:35Optimal model configuration
- 47:21Model pipelines with business logic scripting
- 49:34Real-time spell check for product search
- 50:38Benefits of Triton Inference Server on
- 53:19Join the NVIDIA Inception program for startups
related talks
52mSimplifying the adoption of generative AI for enterprises
20mAccelerating AI pipelines w/ NVIDIA on AWS: An autonomous driving case
43mDeep learning beyond the learning
25mSponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...
32mDeep Learning with TensorFlow, PyTorch and MXNet on AWS
22mQuickly deploy NVIDIA optimized generative AI on Azure AI Studio
35mMLDS 2019: Madhusudan Shekar of Amazon at Machine Learning Developers Summit 2019
23mBest Practices for Building Production-Grade Deep Learning Systems