But Wait! There's...Still More‽ - Observability Data Volumes and Strategies for Managing Them
Talkat Observability Day North America 2024 · on Tuesday, 2024-11-12· at 10:40· Salt Palace | Level 2 | 255 E·
summary
Shows telemetry growth hitting terabytes to petabytes per month and names concrete pain points: storage, write/ingest costs, read/index costs, and ops expertise. Recommends specific mitigations: lower scrape/sample rates, downsampling, retention tiers, label aggregation, and dropping unused series, and cites tools: Prometheus ecosystems (Thanos, Cortex), Prometheus Mimir analyze CLI, OpenTelemetry Collector, Honeycomb/Refinery for tail sampling and Pixie/Pyroscope for profiling. Claims tracing/logs can explode costs and reports these approaches work in practice for lowering volume and cost.
AI-written summary · about
chapters (19)
- 0:00Welcome
- 1:01Speaker overview
- 2:49Observability history
- 4:14Modern observability complexity
- 6:26Metrics and logs explosion
- 8:53Distributed tracing challenges
- 10:16Continuous profiling evolution
- 11:42Storage and cost implications
- 12:47Telemetry processing costs
- 14:13Querying observability data
- 15:36Operational challenges
- 16:19Flawed cost-cutting approaches
- 18:27Metric optimization
- 20:35Dropping unused metrics
- 22:01Log optimization
- 23:46Tracing optimization
- 25:52Profiling optimization
- 26:35Automated solutions
- 27:18Q&A
related talks
16mShrinking the Observability Bill
11mLightning Talk: Beyond Billions: Operating Thanos, Prometheus & OpenTelemetry...
30mPanel: Telemetry That Matters
10mWhy your observability is costing you too much?
27mObservability with Prometheus and beyond
06mObservability Diet: Your 5-Step Plan To Trim the Data Fat
41mScaling Observability: From Chaos to Clarity with Open Source Tools
35mDriving down the cost of observability