Ceph Multi-Site at Scale: Bloomberg’s Disaster Recovery Journey

on YouTube ↗44m

" summary

Bloomberg converts its object store to Ceph multi‑site for server‑side DR, running four clusters (~65 PB raw, ~4,500 OSDs per cluster, >50k buckets, ~1 billion I/O/day). Reproduction uses cosbench and warp plus a homemade verifier on BCC/CICD testbeds; a heavy repro creates ~30,000 buckets and ~10 million objects, originally reproducing after four hours but reduced to ~10–20 minutes. Root cause is a race condition on 4 MB FIFO rados objects (orphaned shards breaking replication); community patches plus a merged branch pass a one‑week stress test.

AI-written summary · about

" chapters (10)

1. 0:00Introduction to speakers
2. 0:44Bloomberg and storage overview
3. 3:54Legacy Ceph clusters limitations
4. 7:50Why multi-site feature was needed
5. 9:15New multi-site architecture
6. 13:56Collaborating with Ceph community
7. 20:59Debugging multi-site replication issues
8. 29:03Testing and deploying fixes
9. 32:35Future plans and acknowledgments
10. 33:17Q&A session