How CERN Monitors the World's Largest Computing Grid with Grafana and Mimir
summary
CERN describes a monitoring pipeline that ingests about 85,000 documents/sec (≈3.3 TB/day) from 200 producers and 15,000 hosts, backs metrics in Kafka/HDFS/Influx and is adding a long‑term Prometheus backend via the Grafana Mimir pilot. The pilot runs on Kubernetes with 29 tenants and ~30 million active series now (capacity ~150M series, ~2,000 queries/sec), integrates with Grafana for alerts (unified alerting migrated), and plans to replace Flume with OpenTelemetry for traces and unified ingestion.
AI-written summary · about
chapters (18)
- 0:00Introduction to CERN
- 2:40The Worldwide LHC Computing Grid (WLCG)
- 3:29CERN's monitoring service
- 4:16Challenges with the monitoring service
- 5:45Current architecture (high level overview)
- 7:09Data types
- 8:55Monitoring service with Grafana (overview)
- 10:26Deployment
- 12:03CERN dashboards
- 12:31Operational challenges
- 14:10Migration to unified alerting
- 16:51Future steps with Grafana
- 18:22Grafana Mimir pilot (a history)
- 19:28Overview of the Grafana Mimir pilot
- 20:16Grafana Mimir pilot deployment
- 20:41Grafana Mimir pilot deployment in numbers
- 21:20More dashboards: Kubernetes cluster monitoring, Grafana Mimir monitoring
- 21:58What's next for CERN with Grafana and Mimir?
related talks
38mEnhancing Ceph Monitoring at CERN: Our Approach and Solutions
37mCloud-native Metric Monitoring with Prometheus
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
36mCilium & Grafana LGTM! - Raymond de Jong
39mHow To Take Prometheus Planet Scale: Massively Large Scale Metrics Installations
24mBanking Observability at Scale
37mM3 and Prometheus, Monitoring at Planet Scale for Everyone
43mBuilding a Multi-tenant Monitoring System with Grafana, Prometheus and Cortex