Global Deduplication for Ceph

on YouTube ↗29m

" summary

A prototype global deduplication system for Ceph computes content fingerprints and maintains a distributed index of object hashes to avoid storing duplicate objects. It is implemented against RADOS and integrated into Ceph's cluster IO paths, tested across five regions, achieving an 8% storage reduction on the sample dataset while describing write redirection, performance impact under load, and observed failure/recovery behavior.

AI-written summary · about

" chapters (5)

1. 0:00Introduction to talk
2. 11:34Global deduplication concept
3. 16:07Ceph deduplication demo
4. 23:23Implementation challenges
5. 27:10Future improvements

" related talks

• 49mIntro to Ceph, the Distributed Storage SystemGregory Farnum · DevConf.CZ 2019
• 02h 59mCeph Storage: a Caffeinated PrimerFederico Lucifredi, Gregory Farnum, JC Lopez · KCD Los Angeles 2024
• 26mImplementing a Resilient Ceph RGW Object Storage with Erasure Coded Data Pool as the Foundation...Siavash Sardari, Vahideh Alinouri, Somayeh Hajahmadi · Cephalocon 2023
• 34mModernizing Ceph Deployments at CERN: CephFS and Object Storage Across Data CentresEnrico Bocchi · Cephalocon 2025
• 29mCeph Operations at ScaleMatt Vandermeulen · Cephalocon 2025
• 35mKeeping Ceph RGW Object Storage ConsistentJane Zhu · Cephalocon 2024
• 21mNew Techniques to Improve Small I/O Workloads in Distributed File SystemsDan Lambright · Vault '19
• 35mMaking Ceph Fast in the Face of FailureNeha Ojha · Cephalocon Barcelona 2019