Implementing a Resilient Ceph RGW Object Storage with Erasure Coded Data Pool as the Foundation...
summary
Digikala builds a Ceph RGW object store on reclaimed hardware with an erasure coded data pool instead of full replication, choosing a 2+4+2 erasure profile after benchmarks that double usable capacity (85 TB replicated vs ~170 TB erasure) using Octopus/Pacific on Ubuntu. Load tests use 80M real images, expect 5000 RPS peak with 8x duplicate requests, and production tuning scales RGW to 9 RGW instances per node with a 2008–2048 threadpool; result: >2× required RPS and capacity but an RGW public network bandwidth bottleneck.
AI-written summary · about
chapters (13)
- 0:00Introduction and team background
- 1:05Problem definition and project goals
- 4:39Current architecture and traffic flow
- 7:15Load testing framework and data set
- 9:51High-level architecture overview
- 11:42Key considerations and decisions
- 13:29Replicated vs erasure coding comparison
- 16:24Erasure coding plugin selection
- 17:53Scaling RGW instances and tuning
- 19:41Data caching strategies
- 21:10Final architecture and optimizations
- 23:04Results and performance metrics
- 24:12Q&A session
related talks
35mKeeping Ceph RGW Object Storage Consistent
02h 59mCeph Storage: a Caffeinated Primer
29mGlobal Deduplication for Ceph
06mNobody Knows What PGs are Good For, Only I Do
49mIntro to Ceph, the Distributed Storage System
34mModernizing Ceph Deployments at CERN: CephFS and Object Storage Across Data Centres
18mScaling RGW D4N Caches With Experimental Offloading
35mMaking Ceph Fast in the Face of Failure