How to Accelerate Model Training and Eliminate the I/O bottleneck for the Cloud
Talkat KubeCon + CloudNativeCon + Open Source Summit China 2023 · on Wednesday, 2023-09-27· at 16:40· 2层 会议室 3 | 2F Room 3·
summary
Describes using object storage with Kubernetes for large-scale AI model training and the storage challenges that arise (low metadata performance, missing atomic rename, eventual consistency). Explains practical techniques at the storage layer—data caching, prefetching, concurrent reads and scheduling—to improve I/O efficiency and enable data to move with containerized compute without changing upper-layer components.
AI-written summary · about
related talks
33mAI Model Distribution Challenges and Best Practices
24mObject Storage Driven Machine Language Workloads
35mWriting Machine Learning Pipelines Against Object Storage
43mProblems and Solutions with AI Model Deployments in k8s
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
47mBuilding AI applications that leverage your data in object storage
31mBuilding a Data Infrastructure for AI/ML
27mHarnessing Kubernetes for Scalable AI/ML Workloads