Improving Business Continuity for an Existing Large Scale Ceph Infrastructure
summary
CERN runs 16 production clusters totaling ~65 PB and expects >100 PB; block storage scales from one to five RBD clusters exposed via OpenStack with storage availability zones. RBD mirroring with journals halves client throughput; snapshot-based mirroring and RBD-to-RBD Cinder backups work well and scale. Object S3 multisite shows up to ~15s replication lag and needs careful resharding; one object cluster is 25 PB with ~2,000 OSDs. Backups use a custom Rust/orchestrated tool handling 850M files and ~6 PB in S3. MDS snapshots hit a Finisher CPU spike (30–40%) that requires pinning or upstream fixes.
AI-written summary · about
chapters (17)
- 0:00Start
- 5:29Outline
- 5:58RBD for OpenStack Cinder
- 7:52RBD Availability Zones
- 10:08RBD Mirroring
- 15:05RBD: BC/DR Summary
- 16:29S3 Object Storage
- 18:19"Traditional" Multisite Configuration
- 20:36Digression on BGP-based Load Balancing
- 21:41Multisite with Sync Policies
- 23:14Multisite with Cloud Sync
- 26:26S3: BC/DR Summary
- 28:55CephFS Snapshots
- 31:30Snapshot Performance Impact (cont'd)
- 33:22cback: Adding Support for Cold Storage
- 34:09Summary on Business Continuity
- 36:09Summary on Disaster Recovery
related talks
28mArchitecting for high availability on Amazon S3
36mHow CERN empowers its users with Kubernetes and OpenShift
29mCloud Foundry Growing Pains: Lessons Learned Keeping the Largest CF Environment Alive and Kicking
39mManaged CF-as-a-Service: An Operator's Cookbook
29mA Multi-Cluster, Multi-Cloud Infrastructure with GitOps at CERN
37mOne Cluster to Rule Them All - Multi-Tenancy for 1k Users
14mKeynote: The Big Leap: Conquering the Legacy System and Building a Scalable Cloud Infra...
24mMigrating Large Scale CF Environments