Merkle Forest, or How We Reduced Metadata Size and Also Speeded Up Duplicate Search
summary
Describes how the TATLIN.BACKUP storage system reduced metadata size and sped up duplicate detection in long-term backup storage by redesigning metadata handling. It explains that metadata growth and frequent duplication were the problems and that they used a Merkle-tree–based approach to deduplicate and compress metadata, improving restore efficiency.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:59Metadata growth problem
- 7:12What is Merkle tree
- 9:47Building Merkle forest in TL Backup
- 13:24Merkle forest results
- 15:37Faster duplicate search
- 20:41Content Defined Chunking
- 26:24Hash function choice
- 31:02Summary and extra properties
- 32:05Q&A
related talks
05mExploring Data Deduplication in LSM Tree-based Key-Value Stores (Work-in-Progress Report)
18mDon't Maintain Twice, It's Alright: Merged Metadata Management in Deduplication File...
24mTSDB Data Compactions on Scale Without Headache: Now & Future
13mClosing the B+-tree vs. LSM-tree Write Amplification Gap on Modern Storage Hardware...
12mThe Dilemma between Deduplication and Locality: Can Both be Achieved?
15mAn Efficient Cloud Storage Model with Compacted Metadata Management for Performance...
20mTiDedup: A New Distributed Deduplication Architecture for Ceph4
15mRemoving Double-Logging with Passive Data Persistence in LSM-tree based Relational...