HadaFS: A File System Bridging the Local and Shared Burst Buffer for Exascale
summary
HadaFS is a deployed burst-buffer file system that bridges local and shared BBs by implementing a Localized Triage Architecture to bind clients to selected bridge servers, a full-path indexing scheme with three metadata synchronization strategies, and a data-management tool Hadash for queries and automated migration. Deployed on the Sunway New-generation Supercomputer, it serves hundreds of applications, scales to 600,000-client scaling and 65,536-process runs, reduces IO folding to boost some workloads by ~30%, and outperforms prior migration tools on mixed-size workloads.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:42Burst Buffer in Typical HPC Storage Systems
- 1:25Challenges with Burst Buffer (3/3)
- 3:31Sample BB System: Sunway New-generation Supercomputer
- 4:59Localized Triage Architecture
- 6:20Namespace and Metadata Handling (2/2)
- 8:24HadaFS I/O Control and Data Flow
- 10:12Data Forwarding Overhead (2/2)
- 12:07Metadata performance evalutaion
- 12:59Data Performance Evalutaion (2/2)
- 14:04Data Migration Evalutaion
- 14:33Interference Avoidance
related talks
12mFacebook's Tectonic Filesystem: Efficiency from Exascale
17mSingularFS: A Billion-Scale Distributed File System Using a Single Metadata Server
22mEveryone Loves File: File Storage Service (FSS) in Oracle Cloud Infrastructure
25mStretch: A Fault-Driven DSM Runtime for Distributed Multithreaded Applications
18mEthane: An Asymmetric File System for Disaggregated Persistent Memory
15mInfiniFS: An Efficient Metadata Service for Large-Scale Distributed Filesystems
19mConsolidating Storage Backends at Scale with Tectonic Filesystem
21mNew Techniques to Improve Small I/O Workloads in Distributed File Systems