The Dilemma between Deduplication and Locality: Can Both be Achieved?
summary
MFDedup implements a classification-based dedupe plus an offline iterative reorganization that detects duplicates mainly against the previous snapshot and rearranges chunks into a compact sequential layout, using Neighbor-Duplicate-Focus (NDF) indexing and an Across-Version-Aware Reorganization (AVAR) scheme. The system keeps indexes for only two versions, nearly eliminates random I/O during restore, cuts GC to near zero, and reports 1.12x–2.19x higher dedupe ratios and 2.63x–11.64x faster restore throughput versus prior work.
AI-written summary · about
chapters (8)
- 0:00Intro
- 1:16The fragmentation problem
- 5:53Example
- 8:44Deletion
- 9:41Backups
- 10:11Deduplication
- 10:49Restore performance
- 12:01Conclusion
related talks
17mLight-Dedup: A Light-weight Inline Deduplication Framework for Non-Volatile...
18mDon't Maintain Twice, It's Alright: Merged Metadata Management in Deduplication File...
20mTiDedup: A New Distributed Deduplication Architecture for Ceph4
05mExploring Data Deduplication in LSM Tree-based Key-Value Stores (Work-in-Progress Report)
26mPhysical vs. Logical Indexing with IDEA: Inverted Deduplication-Aware Index
13mThe what, The from, and The to: The Migration Games in Deduplicated Systems
53mData Deduplication For Redundant Graph Structures Files
17mShieldReduce: Fine-Grained Shielded Data Reduction