⚡ Lightning Talk: Using Decentralized File System to Optimize ML/AI Workloads on Kubernetes
Lightningat Data on Kubernetes Day North America 2023 · on Monday, 2023-11-06· at 17:05· W471ab, McCormick Place·
summary
Describes a decentralized virtual file system (Alluxio) that removes master nodes and uses consistent hashing on the client side to place data and metadata on workers, supporting 30 to 50 million files per worker so a 50-worker cluster caches billions of files. Integrates with Kubernetes via a cloud-native CSI operator and runs end-to-end PyTorch CV training, claiming GPU utilization jumps from 17% to 93% and reduced data loader time from 82% to 1%.
AI-written summary · about
related talks
45mAlluxio Data Orchestration for Machine Learning
27mHow to Eliminate the I/O Bottleneck and Continuously Feed the GPU While Training in the...
20mAdvanced CSI-FUSE Filesystem for AI/ML Data Management in Kubernetes
12mLightning Talk: Distributed AI Without the Infrastructure Tax
28mMaximize GPU Utilization for Model Training
40mOpen Source Memory Speed Virtual Distributed Storage System
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed Caching
32mStreamlining Data Access for with