Lessons From Running Ceph as a Platform at Scale
summary
Sony Interactive Entertainment runs 60 clusters, 250 PB, 30,000 drives, 16 billion objects and finds scale forces concrete solutions: use Ansible + Prometheus + Grafana with recording rules and SLOs to cut noisy alerts; build automation for drive replacement to tolerate ~5% failed drives (~1,500 at once); add an RGW utilization exporter to get per-tenant/user metrics; split massive RBD trees into namespaces to speed up rbd ls; these changes already reduce toil and make incidents more actionable.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 1:46Building a global storage platform
- 3:31Support model and growth planning
- 5:43Managing multiple Ceph clusters
- 7:56Custom metrics and monitoring
- 9:41Handling alerts and SLOs
- 12:33Documentation and automation
- 14:22Drive failure and replacement
- 16:12RBD performance challenges
- 18:16Key lessons learned
- 19:26Q&A session
related talks
11mKeynote: From Busy to Impactful: Maximizing Ceph’s Collective Resources
22mScaling Culture of Resiliency in the Enterprise at Charter Communications
27mSRE Drives Maturity Models
23mScaling Kubernetes in Regulated Environments Through Platform APIs
52mPersistent SRE Antipatterns: Pitfalls On the...
43mSRE From Inception to Global Scale
18mEffective Resource Management: Through Systems Design
53mPanel Discussion - Performance & Scalability