The Butterfly Effect of a Broken Disk: Top-Down Ceph Troubleshooting To Upstream Contribution
summary
Small on-prem Ceph (10 nodes, 10 NVMe, 15 SSDs) experiences severe write latency traced with ceph osd perf, Grafana and dmesg -k to one node where an NVMe repeatedly resets and the OS disk shows EXT4 CRC errors. Kernel softirq storms and ksoftirqd CPU starvation slow every OSD on that node, and synchronous 3‑replication produces huge tail latency—one node (five OSDs) becomes a 25% blast radius. Fix is marking OSDs out and replacing disk; also change unordered hash map to sorted vector with std::sort (PR #67915) to make OSD IDs readable.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:42Ceph 101 and research cluster
- 6:03Top-down detective work
- 8:57Inside Ceph investigation
- 16:37Dying disk froze cluster
- 18:03Incident to upstream contribution
- 22:04Three key takeaways
- 23:05Q&A
related talks
35mOptimising resource allocation in hyperconverged infrastructure
23mHow Obsolete is your Cloud?
33mThe Challenge of Storing Small Objects on a Large Scale
14mKeynote: Building and Sustaining Momentum in Open-Source Storage
34mEvaluating the Performance and Power Efficiency of Ceph Storage Configurations
32mCeph Durability: How Safe Is My Data?
33mDeconstructing an Abstraction to Reconstruct an Outage
29mCeph's journey at SUSE