Lightning Talk: “Metrics That Lie”: Understanding OpenTelemetry’s Cardinality Capping...
summary
OpenTelemetry SDKs enforce cardinality capping (defaults ~2,000) by folding unseen attribute combinations into a single overflow attribute, preventing OOMs but causing queries that filter by attributes to undercount or return zero while the total request count stays correct. Detection is as simple as querying for the overflow attribute; cumulative temporality makes the leak permanent until restart while delta can recover each export cycle. Demo tooling like dash aspire shows an overflow warning during tests.
AI-written summary · about
chapters (10)
- 0:00Misleading OpenTelemetry metrics
- 1:05Metric points and cardinality
- 2:30High cardinality memory issues
- 3:32Cardinality capping mechanism
- 4:54Overflow bucket impact
- 5:57Undercounting real values
- 7:24Detecting overflow attributes
- 8:48Monitoring overflow tools
- 9:30Temporality and cardinality limits
- 9:53Conclusion and best practices
related talks
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
11mLightning Talk: Beyond Billions: Operating Thanos, Prometheus & OpenTelemetry...
53mOpenTelemetry Metrics: observability’s holy grail
30mPanel: Telemetry That Matters
10mLightning Talk: Is OpenTelemetry Too Complicated to Get Started?
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?- 12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
10m⚡ Lightning Talk: Unlocking Customer-Centric Observability: A Case Study of OpenTelemetry To Reduce MTTD < 3 Mins