torchtune: Easy and Accessible Finetuning in Native PyTorch
summary
Shows a working finetuning pipeline for Llama 3 AP on a single A100 that reduces peak GPU memory from >80 GB (naive bf16) to ~58 GB with 8-bit AdamW, to ~48 GB with chunked cross-entropy, then down to ~16 GB by combining nf4 quantization + layerwise compile equals true, and to ~8–40 GB depending on packing and checkpointing choices. Reports throughput from ~2,200 tokens/s baseline to ~5,000 tokens/s and end-to-end training time cuts from ~6 hours to as low as 36 minutes on real datasets.
AI-written summary · about
chapters (6)
- 0:00Introduction to torchtune
- 1:24Design philosophy of torchtune
- 4:14Memory efficiency case study
- 9:50Training speed with torch compile
- 12:21Performance on real datasets
- 15:34Long context techniques
related talks
12mMemory-Efficient AI: How PEFT and PyTorch Enable Accessible LLM Fine-Tuning
27mAccelerating Generative AI
03mFinetune Llama 4 Faster
30mUnderstanding the end-to-end LLM training and inference pipeline
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
09mTailoring Llama for Your Needs: Fine-Tuning & Evaluation
10mFrom Pretrained To Personal: Privacy-First...
35mCreating Your Own LLM Tuning Platform with Open Source Technologies