InftyDedup: Scalable and Cost-Effective Cloud Tiering with Deduplication
summary
A cloud-native system implements deduplication entirely in the cloud, using distributed batch jobs and spot instances to process backup metadata (file recipes and a fingerprint index) and data containers; it deduplicates multi-petabyte backups for a couple of dollars by parallelizing fingerprint comparisons and transferring only missing blocks. Container-aware garbage collection propagates backup expiration to decide profitable rewrites, and a policy-driven hot/cold placement reduces storage costs by 26%–44% while preserving restore latency via per-block expiration and restore-frequency estimates.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:40Motivation for InftyDedup
- 1:11How Cloud Tiering with Deduplication is usually
- 1:41Benefits of Cloud Tiering with Deduplication
- 2:00Limitations of Existing Solutions
- 4:18Deduplication Flow: Filerecipes Uploa
- 5:04Deduplication Flow: Data Upload
- 5:43Why Batch Processing?
- 6:54BatchDedup is scalable
- 8:26Garbage Collection: BatchGC
- 10:20Clouds Offer Multiple Classes of Storage
- 12:27Mixing Storage Classes Benefits
related talks
14mLeveraging software optimization to reduce cloud footprint
01h 00mEconomical and resilient workloads at cloud scale with Veritas (PRT257)
20mRadically simple data protection with AWS and Druva
41mHelp protect your cloud journey with Druva
22mMulti-Purpose Cloud Filesystem
44mStrengthen your cyber resilience and secure cloud data with Veritas
17mBoosting Cost Efficiency by Cost Aware Architecture
21mHybrid Cloud - nothing to worry about! 🌥️