Upstream-First, High Scale Prometheus Ecosystem
summary
Open source telemetry called telemeter collects anonymized cluster signals — counts of updates per cluster, update errors, progress, component health and platform name — by deploying Prometheus Operator with HA Prometheus, Alertmanager and Thanos in every cluster. An initial federated scrape every 4.5 minutes used an in‑memory hash and failed under high volume, so the system moves to Prometheus remote_write into a Thanos receiver (push model), adds object storage for long retention, and exposes data via Thanos Querier and Loki for logs.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 0:45Telemeter
- 2:22Build Telemetry
- 2:43Prometheus Operator
- 3:43Prometheus Deployment
- 6:07Niv Stack
- 7:56Reference Architecture
- 8:25Lessons Learned
- 9:17Outro
related talks
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry
41mHigh Available + Scalable Prometheus with Thanos in Alibaba
26mPrometheus That Scales
27mObservability with Prometheus and beyond
25mMulti-Cluster Observability: What you need to survive with a fleet
31mHandling Billions of Metrics with Prometheus and Thanos
24mObservability at Scale: How Open Systems Collects Telemetry from Over 10'000 Edge Devices Worldwide
01h 19mIntro + Deep Dive: Prometheus