Overcoming I/O Bottlenecks in LLM Training with Open-Source Distributed Caching
Talkat Open Source Summit Europe 2023 · on Thursday, 2023-09-21· at 15:55· Room 0D-2-0D-3 (Floor 0), Abandoibarra Etorb., 4, 48011 Bilbao, Bizkaia, Spain·
summary
An open-source distributed caching layer (Alluxio) is presented that sits between object/HDFS storage and training nodes, using hierarchical cache: memory → local disk page cache → remote cache cluster to cut repeated I/O. Measured wins: data loading drops from 82% of job time and GPU utilization rises from <20% to much higher; model deploy latency falls from ~15 minutes to ~3 then <1 minute. Reads show positional reads beat streaming for large Arrow blocks (4 KB random reads) while streaming wins for many small files.
AI-written summary · about
chapters (7)
- 0:00Alexio and data caching
- 3:10Alexio’s role in AI
- 5:01Traffic patterns and caching
- 10:22Hybrid multi-cloud caching
- 16:00Real-world caching strategies
- 21:40Future of AI caching
- 23:29Q&A
related talks
38mExploring Distributed Caching for Faster GPU Training with NVMe, GDS, and RDMA
28mMaximize GPU Utilization for Model Training
27mHow to Eliminate the I/O Bottleneck and Continuously Feed the GPU While Training in the...
45mAlluxio Data Orchestration for Machine Learning
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
26mUnlocking Scalable Distributed Training With Arrow Data Cache on Kubernetes
26mSpeed up Your ML Workloads With Kubernetes Powered In-memory Data...
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...