Fast Cloud Storage for AI Jobs via Grouped I/O API with Transparent Read/Write...
summary
AITURBO is a cloud storage system that cuts AI checkpoint IO by using the grouped I/O API to collect per-tensor metadata, derive an optimized write plan (bilinear program solved by binary‑searching T and LP checks), and execute many‑to‑many transfers over the 200 Gbps compute fabric while using host DRAM as a fast tier. It detects duplicates via Blake3 checksums, sustains required bandwidth reductions (example: 86 GB/s needed for a 38B model), runs on 64 Ascend 910B NPUs, plans in seconds–minutes, and outperforms Megatron and Gemini in production at Huawei Cloud.
AI-written summary · about
chapters (12)
- 0:00Air Turbo for AI jobs
- 1:24Checkpoint IO challenges
- 3:30AI job IO principles
- 4:13Deduplication for IO payload
- 7:03Group IO API design
- 8:47Checksum deduplication
- 9:48Compute storage fusion
- 11:12Automatic IO optimization
- 13:43Evaluation setup
- 14:23Checkpoint performance
- 15:48Air Turbo improvements
- 16:08Conclusion
related talks
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
12mOptimizing AI Data Infrastructure with F5 and
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
14mArchitecting Cloud Storage for AI Native Applications
37mHow to Accelerate Model Training and Eliminate the I/O bottleneck for the Cloud
27mHow to Eliminate the I/O Bottleneck and Continuously Feed the GPU While Training in the...
04mA Cloud Native Stack From Bare Metal To Tokens for Large-Scale AI Inference
29mAccelerate your cloud architectures with the right storage options