Changing the Engine Mid-Flight: Zero-Downtime Ceph Upgrades
summary
Upgrades a 10–50 node bare‑metal Ceph cluster serving 5G core (AMF/UPF) from Reef→Squid (v18→v19) with zero downtime by using a rehearsed sequencing framework and checklist. Covers three real failures: monitor quorum loss with root‑cause and recovery steps, OSD rebalance storms that threaten stability, and incompatible client versions that silently block the path. Uses MGR→MON→OSD→MDS/RGW ordering, monitor DB backups, safety flags (noout/norebalance), limited parallel OSD upgrades and live benchmarks; upgrade completes without service interruption.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:09Why upgrade Ceph in production
- 2:55Upgrade challenges and limits
- 4:28Version requirements and compatibility
- 7:09Upgrade architecture and plan
- 8:16Cluster prep for upgrade
- 10:12Updating Ceph manager service
- 12:32Updating Ceph monitors
- 14:08Zero-downtime OSD upgrade
- 17:34CephFS upgrade steps
- 20:57Post-upgrade cluster checks
- 22:51Key takeaways
- 25:25Q&A
related talks
30mWhatever Can Go Wrong, Will Go Wrong – Rook/Ceph and Storage Failures
27mCeph, the future of Storage
46mEnd-Users Won't Even Notice, Painless Kubernetes Upgrades
35mBest Practices of Production-Grade Rook/Ceph Cluster
46mKubernetes Stateful Application Disaster Recovery with Zero Data Loss
25mAutomating Kubernetes Upgrades for Zero Downtime
35mDeploy Ceph in Kubernetes: Rook Deep Dive
30mWhen the Edge Can’t Afford a Third Node: A Storage Solution for Two-Node Kubernetes C...