Enhancing Observability and Monitoring for Large Ceph Clusters at Scale
summary
Proposes a four-layer observability stack with nearby and remote producers, a NATS messaging backbone, and lightweight consumers that perform on-demand and event-triggered collection so long-running full-log ingestion is avoided. Proof-of-concept in Kubernetes shows NATS handling high message rates and reducing stored volume, but hits practical limits: RadosGW exposes only Kafka-style bucket notifications and HTTP with limited fields, Rook prevents easy sidecars, and many metrics (about more than 1000 metrics) plus one-hour updates for usage force selective capture. The approach is promising but not finished.
AI-written summary · about
chapters (8)
- 0:00Introduction to talk
- 1:07What is observability
- 6:14Problems with traditional logging
- 10:36Event-driven design solution
- 16:47Producer-consumer architecture
- 21:07Types of monitoring needed
- 24:53Future directions and AI
- 28:51Q&A session
related talks
31mMonitoring and Centralized Logging in Ceph
22mReworking Observability In Ceph
01h 02mDesigning an open-source observability stack for Kubernetes
38mEnhancing Ceph Monitoring at CERN: Our Approach and Solutions
40mBuilding a Scalable Observability Platform for Edge Compute Nodes
23mWhen Kafka Goes Cloud Native: Observability That Actually Works!
37mEnhancing Observability in Rook Ceph
50mIntuit Journey To Unified Observability at Scale: Challenges, Be...