TierTrain: Proactive Memory Tiering for CPU-Based DNN Training
summary
TierTrain implements a proactive queuing daemon that uses PyTorch hooks and a first-epoch profile to evict and prefetch saved tensors across HBM/DRAM, CXL-attached memory and NVMM/Optane based on measured idle times and bandwidth. Evaluation on real CXL and Optane systems shows 59–83% average fast memory reduction and 1–16% performance overhead, with up to 25–74% peak reduction and 35–84% faster training than existing tiering under tight memory.
AI-written summary · about
chapters (15)
- 0:00TierTrain introduction
- 1:04Memory costs and wall problem
- 2:06Memory tiering background
- 3:13DNN training access patterns
- 5:04Exploiting idle time
- 6:50Eviction and prefetching
- 8:16Results on CXL memory
- 9:20DRAM and Optane results
- 10:02Insufficient DRAM handling
- 11:29Conclusion and benefits
- 12:21Q&A: GPU applicability
- 13:28Q&A: Memory bus limits
- 14:59Q&A: Rematerialization interaction
- 16:07Q&A: Memory traversal patterns
- 17:34Q&A wrap-up
related talks
14mHyCache: Hybrid Caching for Accelerating DNN Input Preprocessing Pipelines
19mML training acceleration with heterogeneous resources in Bytedance
12mBehemoth: A Flash-centric Training Accelerator for Extreme-scale DNNs
28mMaximize GPU Utilization for Model Training
15mColocating ML Inference and Training with Fast GPU Memory Handover
26mMemory-Efficient LLM Inference on Edge Devices With NNTrainer
12mMemory-Efficient AI: How PEFT and PyTorch Enable Accessible LLM Fine-Tuning
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective