Observability at Scale: How Open Systems Collects Telemetry from Over 10'000 Edge Devices Worldwide
Talkat Observability Day Europe 2023 · on Tuesday, 2023-04-18· at 14:45· Hall 7, Room E | Ground Floor | Europe Complex·
summary
Open Systems monitors a fleet of over 10,000 edge devices and central clusters, ingesting roughly 110 million metrics (bursting to 240M) and the uploader reports ~5 TiB of logs per day, using Thanos receiver and Grafana Loki backends. Architecture uses nginx with client TLS and tenant routing into dedicated Thanos pipelines, a telegraph producer buffer on devices to protect Prometheus, and Thanos 0.29’s active series limiting to avoid cardinality explosions. Alerts flow through Alertmanager to a Kubernetes operator called alert Handler for enrichment and grouping; the pipelines run in production and are reducing scale failures while alert-noise work continues.
AI-written summary · about
chapters (22)
- 0:00Observability at scale intro
- 1:03Open Systems and SASE model
- 2:25Edge devices growth
- 3:26Mission Control monitoring
- 4:07Current monitoring pipeline
- 5:34Metrics stack evolution
- 6:40Scaling observability challenges
- 7:22Action plan for scalability
- 8:26Project timeline
- 8:47Thanos for metrics pipeline
- 10:51Why Thanos fits edge devices
- 11:12Global metrics architecture
- 12:34Tenant-based ingestion pipeline
- 13:16Cardinality explosion mitigation
- 14:21Prometheus Telegraf edge metrics
- 15:42Global metrics reading
- 16:24Query quality of service
- 17:26Portal query prioritization
- 18:09Unified observability state
- 18:52Alertmanager replacement
- 20:19Alert Handler empowerment
- 21:03Conclusion and future goals
related talks
10mUpstream-First, High Scale Prometheus Ecosystem
40mBuilding a Scalable Observability Platform for Edge Compute Nodes
25mMulti-Cluster Observability: What you need to survive with a fleet
06mFrom Sensors to Insights: Building an Open Observability Pipeline for Industrial Systems
31mHandling Billions of Metrics with Prometheus and Thanos
11mLightning Talk: Beyond Billions: Operating Thanos, Prometheus & OpenTelemetry...
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments
38mEnhancing Ceph Monitoring at CERN: Our Approach and Solutions