Upgrade and Scale-Out an In-production Ceph Cluster on Mixed Arm Micro-server Platform
summary
Case study of an in-production Ceph+OpenStack cluster: starts with 21 OSDs serving ~140 VMs, cluster peaks ~7,000 IOPS with per-disk averages ~100 IOPS so performance is I/O bound while CPU stays ~40%. Short-term fix adds 15 OSDs to cut per-OSD IOPS; long-term adds 64-bit nodes and migrates software. Migration sequence upgrades monitors first, then marks out one OSD at a time, runs ~36 loops with an automation playbook, taking ~108 hours total and no service impact. Key changes: Jewel→Luminous/BlueStore, Debian→CentOS, mixed 32/64-bit ARM.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:26Ambedded Background & Product
- 2:54Mars 400 IU 8 nodes ARM Micro-Server Cluster
- 4:05Customer OpenStack Cloud
- 6:576 Months after production
- 8:43Log Observations
- 12:10CPU Usage & CPU Load
- 13:17The Cause
- 15:04The Initial Ceph Cluster
- 16:36The Short Term Solution
- 18:39Long Term Solution
- 21:07Plan the IOPS
- 24:00Our Challenges
- 25:51Challenges: Upgrade and Scale-Out the Mixed-Cluster
- 26:44The Standard Upgrade Procedure
- 27:15Options of Upgrade MON and WEB UI
- 28:48Upgrade Monitors Steps
- 29:30Critical Step - Migrate the Monitors
- 32:39Ceph Version
- 32:58Migrate the OSDs
- 35:24OSD Migrate Procedure with Ansible Playbook
- 37:27The Privileges of Micro-Server Architecture