GPU Training Made Easy-Pre-train & fine-tune NLP models
summary
Demo uses minGPT on PyTorch with four 2080 Ti GPUs to pretrain a character-level model on ~1.1M Shakespeare characters (65 unique chars); configured model is 8 layers × 8 heads (~25M params). Practical tools shown: nvidia-smi, torch.cuda memory_stats, empty_cache and gc.collect; training with batch size 512 runs but raising to 1024 triggers a CUDA OOM (batch size 512 vs 1024 OOM). A TrainML trial link and notebook are provided.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 3:50Where to start
- 6:52Data
- 8:42Model Size
- 9:51Training Script
- 10:44CPU Utilization
- 12:31Hstop
- 14:25Nvidia MSI
- 16:21Memory Management
- 17:36Infrastructure
- 19:54Memory Stats
- 21:57Memory Footprint
- 25:43Batch Size
- 27:56GPU vs Spark
- 29:52TrainMLai
- 31:40Free Trial
- 34:30QA
- 36:20Best Practices
related talks
28mTraining Large Language Models on Kubernetes
02h 50mUnderstanding and Using AI Language Models
13mPreview: ML Training
01h 40mA Deep Dive into NLP with PyTorch
43mState of GPT
19mBeat GPT 4 & Build the Best AI For You
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
01h 00mInside GPT – Large Language Models Demystified