Ceph Operations Retrospective: Lessons from Scaling and Managing a Multi-PB Cluster

on YouTube ↗34m

" summary

Ceph Pacific cluster uses CephFS and RGW, grows from 4.5 PB to ~9 PB raw across ~300 OSDs and adds nodes of ~500 TB each, ingesting ~40–55 TB/week; metadata lives on HDD with two NVMe used for WAL/DB, causing performance and failure modes. A multi‑MDS setup hits a race with frequent unlinks and snapshots so the team falls back to single MDS fix and later runs a journal flush + first‑damage script to remove ~10 corrupted objects and restore service. Expansion proceeds with slow backfill (2–5 GB/s) and incremental PG increases (4→128) over days to avoid client impact.

AI-written summary · about

" chapters (10)

1. 0:00Introduction and background
2. 1:47Use case and hardware design
3. 7:03Initial setup and challenges
4. 10:38Backup issues and snapshots
5. 13:49Cluster updates and MDS failure
6. 20:53Metadata damage and recovery
7. 23:03Expanding the cluster
8. 27:56Performance and PG count issues
9. 30:47Lessons learned and future plans
10. 31:57Q&A