InfiniFS: An Efficient Metadata Service for Large-Scale Distributed Filesystems
summary
InfiniFS implements access-content decoupling to partition directory metadata for both locality and load balance, a speculative path resolution that predicts directory IDs and issues parallel lookups to cut path latency by about 74%, and an optimistic access metadata cache that validates caches on servers to avoid near-root invalidation storms. On 32 servers it sustains up to 3.3 million file-creates/sec, reports 18× higher directory-read and 4× higher file-create throughput versus prior systems, and supports directory trees up to 100 billion files.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:27Outline
- 1:08Filesystem Metadata
- 2:01Challenges of efficient Metadata Service
- 5:03InfiniFS Architecture
- 5:46Key Designs
- 8:122. Speculative Path Resolution
- 12:35Scalability
- 13:17Breakdown
- 14:28Conclusion
related talks
17mSingularFS: A Billion-Scale Distributed File System Using a Single Metadata Server
21mNew Techniques to Improve Small I/O Workloads in Distributed File Systems
20mscoutfs: Large Scale POSIX Archiving
22mEveryone Loves File: File Storage Service (FSS) in Oracle Cloud Infrastructure
29mAccelerating Filesystem Checking and Repair with pFSCK
16mHadaFS: A File System Bridging the Local and Shared Burst Buffer for Exascale
28mAsynchronous Directory Operations in CephFS
12mFacebook's Tectonic Filesystem: Efficiency from Exascale