OSD Down! but Why?
summary
Step‑by‑step method for a crashed OSD: check the cluster log or run seph‑w / seph health for immediate errors, then open the OSD log, find the abort/stack trace and use thread ID search to follow that thread back to the root event (often a BlueStore checksum or assert). If logs lack detail, increase log level (CPU/memory cost) temporarily, reproduce, then search tracker and use match source version plus git grep on the repo to map file/line to code and confirm whether it’s a known buffer corruption or needs a bug report.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:42Why OSDs go down
- 1:46Initial troubleshooting steps
- 2:28Debugging OSD crashes
- 3:54Reading OSD logs
- 6:46Log levels and debugging
- 8:12Stack traces and assertions
- 10:00Searching for solutions
- 11:29Analyzing source code
- 13:16Debugging placement groups
related talks
24mAverting crisis at multi-petabyte scale
20mPer OSD Recovery Bandwidth Control Based on dmClock
36mHow to Debug an Asynchronous Odyssey Without Attracting the Orderlies’ Attention
25mWe're Still Down: A Metastable Failure Tale
37mMessage processing failed! But what's the root cause?
35mMaking Ceph Fast in the Face of Failure
41mFinding the Needle in the Haystack or Troubleshooting Distributed Systems
49mImproving Observability of Distributed Systems