Common Support Issues And How To Troubleshoot Them
summary
Ceph flags a slow request when an I/O exceeds the 30s default and common causes here are overloaded disks (iowait reported up to ~600ms, CPU 80–90%), network issues (MTU/jumbo frame mismatches, retransmits), OSD recovery/churn and LevelDB compaction/splitting on large buckets. Useful diagnostics: ceph health detail, ceph osd tree, ceph log parser (GitHub), iostat, perf top, ifconfig and ping, and checking deep-scrub timings. Immediate fixes include increasing OSD timeouts and disabling background compaction for a pool (no_recovery/no_scrub), offline compaction, and moving index pools to SSD/NVMe; long term: shard buckets (~100k objects/shard) or migrate to RocksDB.
AI-written summary · about
chapters (13)
- 0:00Start
- 4:15Configuration Issues
- 4:48Deep Scrubbing
- 5:37Waiting for Rewrite Locks
- 6:18Waiting for Sub Ups
- 7:49Common Troubleshooting Techniques
- 9:03Verifying this Performance on Os
- 9:52Network Performance
- 15:51Bucket Sorting
- 16:53Possible Causes
- 20:12The Temporary Solutions
- 23:47Offline Compaction
- 26:50Summary
related talks
38mEfficient Ceph Performance Troubleshooting in Production Using eBPF
34mDay 2 Operations : Make Friends with Your Ceph Cluster
39mUsing EBPF to Develop a Powerful Tool for Diagnosing Ceph Performance Issues
31mMonitoring and Centralized Logging in Ceph
38mEnhancing Ceph Monitoring at CERN: Our Approach and Solutions
22mReworking Observability In Ceph
44mCeph Crash Telemetry - Observability in Action
58mTroubleshooting your Elasticsearch cluster like a Support Engineer