Enhancing Ceph Monitoring at CERN: Our Approach and Solutions
summary
CERN migrates monitoring from Graphite to Prometheus + Thanos since 2018, storing five years of metrics with 15s raw resolution and automated downsampling on a two-node Thanos compactor; about 100 PB across ~20 clusters and 11,000 servers feed node_exporter plus custom scripts. Alertmanager routes by severity to email, ServiceNow, Mattermost or phone. Client-facing gains include per-client CephFS session metrics that identified MDS OOMs from cap-release spikes and enabled live mitigation, and S3 multi-site byte-diff checks plus OpenStack exporter joins to show tenant/user context. Monitoring is duplicated across two data centers with manual failover.
AI-written summary · about
chapters (8)
- 0:00Introduction to CERN and SEF
- 4:35Evolution of monitoring at CERN
- 11:40Host and hardware monitoring
- 15:30RBD and S3 multi-site monitoring
- 17:35CephFS client visibility and debugging
- 22:55Ceph outages and lessons learned
- 27:28Ceph science user group
- 28:09Q&A session
related talks
31mMonitoring and Centralized Logging in Ceph
33mMonitoring Ceph with Prometheus
26mPrometheus That Scales
29m“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus
10mUpstream-First, High Scale Prometheus Ecosystem
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments
25mHow CERN Monitors the World's Largest Computing Grid with Grafana and Mimir
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry