TiDedup: A New Distributed Deduplication Architecture for Ceph4
summary
TiDedup introduces a Ceph-integrated deduplication design that claims up to 34% data reduction, improves foreground I/O throughput by about 50% during deduplication, and cuts reference-scan time by more than 50%. It implements three concrete mechanisms: selective cluster-level crawling to avoid metadata bloat, an event-driven tiering with content-defined chunking to avoid long background locking and fixed-size chunks, and a shared reference back pointer to make snapshot-aware reference correction fast and distributed.
AI-written summary · about
chapters (6)
related talks
12mThe Dilemma between Deduplication and Locality: Can Both be Achieved?
18mDon't Maintain Twice, It's Alright: Merged Metadata Management in Deduplication File...
21mData Domain Cloud Tier: Backup here, backup there, deduplicated everywhere!
05mExploring Data Deduplication in LSM Tree-based Key-Value Stores (Work-in-Progress Report)
11mAdapting TiKV for Cloud Storage
17mShieldReduce: Fine-Grained Shielded Data Reduction
26mPhysical vs. Logical Indexing with IDEA: Inverted Deduplication-Aware Index
18mTiFlash: Make TiKV 10x Faster and HTAP-able