From Chaos To Clarity: Systematic Ceph Cluster Analysis for Rapid Troublesho...
summary
A systematic pipeline ingests Ceph diagnostic bundles, scans releases against a YAML of notorious bugs, and flags imbalanced pools, hot-primary OSDs, cross-rack replica risks, MDS create/unlink latency, and RGW shard-object ratios; a CLI single-binary analyzer called Dr. Octopus runs deeper checks and executes fixes like upmap remap to move PGs (used on a 5 PB RGW subcluster), a web front end at analyzer.cluster.com and a config diff showing 2,147 configs in 19.2.3.
AI-written summary · about
chapters (22)
- 0:00Introduction and overview
- 1:05Analyzing Ceph cluster health
- 3:12Checking Ceph versions and critical bugs
- 3:32Data balancer and PG distribution analysis
- 4:38Cross-row analysis and spreading pools
- 5:21OSD performance analysis
- 6:04CephFS MDS latency and client issues
- 7:09RGW bucket and sharding analysis
- 8:56Cluster analysis report and suggestions
- 9:17Introducing the Dr. Octopus CLI tool
- 10:43Using Dr. Octopus for day-to-day operations
- 12:48Web-based analyzer tool overview
- 13:10CLI tool features and recommendations
- 14:16PG distribution visualization
- 15:00Fixing OSD full warnings with upmap
- 15:43Future plans for balancer improvements
- 16:25Terminal dashboard for Ceph
- 17:06Analyzer tool statistics and trends
- 18:30Version score distribution insights
- 19:38Config diff and configuration viewer
- 21:22Open-source tools and resources
- 21:46Q&A session
related talks
38mTesting Ceph: Pains & Pleasures
32mStretch Clusters in Ceph: Algorithms, Use Cases, and Improvements
35mKeeping Ceph RGW Object Storage Consistent
02h 59mCeph Storage: a Caffeinated Primer
49mIntro to Ceph, the Distributed Storage System
25mSupporting 3 Availability Zones Stretch Cluster
25mExplicitly Supporting Stretch Clusters in Ceph
37mNew Read Balancer in Ceph