Scaling Telemetry To Terabytes: Applying Google's Principles To OpenTelemetry...
summary
Describes running a planet‑scale time‑series DB Monarch that holds ~100 billion series and ~800 TB in memory, and moving from pull to a push client to avoid hotspotting and lower cost. Presents a control plane that tells clients which metrics to send, an automated metric turndown that disables unused series, a finding of one metric defined 10,000 times causing massive derived‑data waste, and an OpAMP policy proposal that prototypes show can cut collector pipeline costs.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:04Google’s telemetry architecture
- 3:19From pull to push clients
- 8:17Centralized cost control
- 13:14Scaling client collection
- 17:29Optimizing control plane
- 19:14OpenTelemetry & OpAMP
- 20:41Key lessons
- 21:47Q&A
related talks
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
30mPanel: Telemetry That Matters
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
27mThe Signal in the Storm: Practical Strategies for Managing Telemetry Overload
11mLightning Talk: Beyond Billions: Operating Thanos, Prometheus & OpenTelemetry...
30mBut Wait! There's...Still More‽ - Observability Data Volumes and Strategies for Managing Them
35mOpenTelemetry for Busy People