Taking Control of Metrics Growth and Cardinality: Tips for Maximizing Your Observability
summary
Cloud-native metric volumes explode: a 10-endpoint precloud setup (~150,000 time series) can become 150 million time series when pods scale to 30,000 and experiments multiply labels. Practical controls include using m3 and the Prometheus data model, relabel rules and rollup/downsampling policies, clamping emergency emitters, and mirroring telemetry for safe experiments. Instrumentation rules matter: record HTTP route not full URL and drop user_id from metrics. Staffing guidance: dedicate ~1–2% of engineers to platform work, expect metrics infra to be several percent of ops spend.
AI-written summary · about
chapters (8)
- 0:00Intro
- 2:47Our mission help customers get to remediation as quickly as possible
- 5:43Growth in monitoring data at Uber
- 11:31Scenarios for taming data growth and cardinality
- 12:18Tips for how to reduce these tensions
- 16:22Tips for managing metrics at a more macro level
- 18:40Tips for how to make observability a team effort
- 22:40Internal KPIs and metrics - meta metrics
related talks
18mArchitecting Growth: Scaling Tactics for Prometheus Metrics...
16mShrinking the Observability Bill
20mThe Sins of High Cardinality
37mCloud-native Metric Monitoring with Prometheus
30mBut Wait! There's...Still More‽ - Observability Data Volumes and Strategies for Managing Them
39mThe Ticking Timebomb of Observability Expectations
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
25mUnbreakable Prometheus: Leveraging Limits for Consistent Monitoring