Uncovering Access, Reuse, and Sharing Characteristics of I/O-Intensive Files
summary
Analysis of Cori production logs (84M Darshan records, 52M files, ~13 PB total) uses Darshan and Lustre metadata to isolate I/O‑intensive files (>=100 GB and >=2 accesses), leaving ~8 PB across ~400k jobs. Findings: 71% of hot files are read/write, 22% read‑only, 7% write‑only; over 80% of runs perform only reads or only writes; producer–consumer interarrival averages ~31 hours; 80% of runs move <12 MB but mean reads/writes are 17/25 GB; strong Lustre OST imbalance and intra‑run rank I/O time variance up to ~50 s.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:24HPC systems suffer from severe I/O bottleneck!
- 1:40Our Study Platform
- 5:26Terms & Definitions
- 10:32Read & Write Run Inter-arrival Times
- 11:48Consecutive Runs of the Same Type
- 12:52File Sharing Across Applications
- 14:58Data Transfer Per Read & Write Run
- 16:27Data Transfer Per OST
- 18:05Intra-run I/O Variability
- 22:42Conclusion
related talks
01h 06mBest practices when accessing Big Data or any other data!
20mFull Lifecycle Data Analysis on a Large-scale and Leadership Supercomputer...
25mEnd-to-end I/O Monitoring on a Leading Supercomputer
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...
52mApache Iceberg - An Architectural Look Under the Covers
43mOptimizing Ceph IO for High Throughput Particle Physics Workflows
34mWhat We Talk About When We Talk About Disk IO
36mRapid Processing of NASA Satellite Data Stored with Ceph