CRUSH-ing the OSD Variance Problem
summary
Manages a Ceph cluster that grows from 1,500 to about 5,000 OSDs (≈40 PB raw) while maintaining >50% average utilization, and faces wide OSD-utilization variance that limits usable headroom. Traditional reweight-by-utilization runs move petabytes over months and only shave standard deviation slowly; deploying the upmap balancer shrinks standard deviation from ~8–11 to ~2 in days, freeing roughly 6 PB of usable capacity. Adding OSDs exposes CRUSH collision edge cases that trigger unexpected PG moves, so lightweight tools to track PG backfills and predicted OSD fill rates prove essential.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 4:59Understanding CRUSH and OSD variance
- 10:21Challenges with reweight by utilization
- 19:35Introducing upmap balancer
- 26:46Scaling the cluster with new OSDs
- 31:40Unexpected placement group movement
- 38:00Adding OSDs with upmap
- 40:11Conclusion and future outlook
related talks
28mA Welcoming Ceph Experience Improvements for First-Time Users or: How to Make Beginners Love Ceph
19mBeyond Balancing: How Upmaps Were Used To Avoid Disaster!
10mCluster-scale Problem Solving
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
55mCVE Overload: Scaling Container Vulnerability Scans
31mMonitoring and Centralized Logging in Ceph
27mSWITCH: Operations, Data Management and Automation
12mUsing rbd Migration Tooling in Our Ceph Operations