Beyond Balancing: How Upmaps Were Used To Avoid Disaster!
summary
Thousands of NVMe servers show a kernel-boot fault: specific rettimer cards in slot groups die from an overvoltage at reboot, risking many PGs losing multiple replicas. The fix uses Ceph upmap remap entries to move PG replicas off vulnerable OSDs, script the changes, disable the balancer, and then reenable it when safe. A custom rettimer_upmap tool automates limits to size−min_size per failure domain, validates by simulating OSD stops (zero unsafe PGs) and prevents a multi-cluster outage.
AI-written summary · about
chapters (8)
- 0:00Introduction and speaker background
- 1:47Ticking time bomb in storage
- 5:19Root cause of drive failures
- 7:27What are upmaps and why they're awesome
- 9:55Visualizing the problem with upmaps
- 11:59Rettimer upmap tool creation
- 13:45Other uses for upmaps
- 17:50Q&A session
related talks
44mCRUSH-ing the OSD Variance Problem
50mOps issues
28mA Welcoming Ceph Experience Improvements for First-Time Users or: How to Make Beginners Love Ceph
40mJuggling Petabytes: Managing Ceph at Scale with Ceph-ansible
21mSponsored Session: Fault-Tolerant Training: How We Build Rel...
37mStop Hitting Yourself!
33mPreserving IOMMU States During Kexec Reboot
15mKeeping a Hypervisor Fleet Up to Date with Minimal Customer Disruption