Day 2 Operations : Make Friends with Your Ceph Cluster
summary
Assumes a healthy cluster and then gives concrete operational rules: aggregate logs with ELK stack and tune log levels; collect metrics with Prometheus + Telegraf or the Ceph metrics project and feed Grafana dashboards; use configuration management with GitOps and run it regularly; an upgrade once doubled write throughput but a missed luminous flag causes OSD flapping that eats RAM; replace OSDs using either two‑rebalance or single backfill procedures; target 100 PGs per OSD (start low and increase); scrubs default weekly and must be windowed to avoid client impact; new Nautilus features—balancer, orchestrator, dashboard, PG auto‑tuning—reduce manual toil.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:36logging & monitoring
- 1:48Tens, hundreds, thousands components on a lot of machines
- 5:36monitoring vs metrics
- 6:05monitor using your tools
- 7:11metrics : use the right tool for you (and Grafana)
- 8:17mind major and minor ceph versions, statistics
- 10:18please, use a ceph management tool
- 11:42run them regularly
- 16:52release notes are your go-to reference
- 17:31follow them rigorously, up to the last word
- 21:12Run SMART checks
- 24:37PGs are your most precious resource
- 25:56plan carefully
- 26:51start low, increase incrementally
- 28:27essential to data integrity
- 28:48complementary with bluestore checksums
- 32:50balancer mgr module orchestrator mgr module dashboard PG decrease and automatic tuning disk failure prediction centralized configuration in monitors
related talks
31mMonitoring and Centralized Logging in Ceph
28mCommon Support Issues And How To Troubleshoot Them
38mEnhancing Ceph Monitoring at CERN: Our Approach and Solutions
44mCeph Crash Telemetry - Observability in Action
38mEfficient Ceph Performance Troubleshooting in Production Using eBPF
32mEnhancing Observability and Monitoring for Large Ceph Clusters at Scale
35mFrom New Cluster to Insight. Deploying Monitoring and Logging to Kubernetes
22mReworking Observability In Ceph