The Signal in the Storm: Practical Strategies for Managing Telemetry Overload
summary
Argues that turning raw traces, logs and metrics into entities as first-class signal plus explicit relationships cuts telemetry from gigabytes to kilobytes per customer by design, enabling aggressive sampling, local storage and on-edge querying. Implements a prototype in the collector and a productized flow using OpenTelemetry and eBPF auto instrumentation (Grafana eBPF), keeps PII local, and reports it works in production with major cost, scale and privacy gains while acknowledging added distributed-query complexity.
AI-written summary · about
chapters (6)
- 0:00Introduction and challenges
- 1:45Telemetry collection methods
- 4:44Ontology-first approach
- 8:34Leveraging entity signals
- 14:55Distributed edge processing
- 19:25Practical implementation
related talks
02h 41mThe Science of Signals: Mastering Telemetry for Observability
05mTelemetry Pipelines: Your Recipe for Success
30mPanel: Telemetry That Matters
01h 10mHow to Effectively Spy on Your Systems
52mHow to effectively spy on your systems
35mOpenTelemetry for Busy People
22mSeeing Through the Chaos of Modern Systems
24mScaling Telemetry To Terabytes: Applying Google's Principles To OpenTelemetry...