Deep Linking Metrics and Traces with OpenTelemetry, OpenMetrics and M3
Talkat KubeCon + CloudNativeCon North America 2019 · on Thursday, 2019-11-21· at 10:55· Room 11AB - San Diego Convention Center, Upper Level·
summary
Claim: metric datapoints carry trace IDs as exemplars in OpenMetrics so dashboards can jump straight to the exact trace. Implementation uses OpenTelemetry exporter to attach trace IDs, Prometheus scraping, remote write into Prometheus -> M3 storage and Jaeger for traces, with Grafana showing trace IDs next to points. Demo works end-to-end and shows a 54k→242k Grafana Prometheus installs stat; caveats: tracing storage and sampling matter (Slack samples 1%), upstream Prometheus client/server changes are still in progress.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 7:43Metrics and Traces
- 12:27Demo
- 15:25OpenTelemetry
- 19:52Prometheus M3
- 23:02Metrics aggregation
- 25:39Project Status
- 27:24Thank You
- 28:04Embedded Traces
- 29:32Cardinality
- 31:11Unique combinations of metrics
- 32:38Inference DB
- 33:51Prometheus
related talks
13mConnecting Prometheus and OpenTelemetry Data for Faster Troublesho...
37mM3 and Prometheus, Monitoring at Planet Scale for Everyone
21mImproving Monitoring Systems Interoperability with Prometheus & OpenMetrics
17mOpenTelemetry Looks Good To Me: Loki, Grafana, Tempo, Mimir
41mModern Application Debugging: An Introduction To OpenTelemetry
09mGraduated Project Lightning Talk: Using Trace Data for Monitoring & Alerting of Appl...
28mMonitoring Microservices w. Prometheus & Tracing Apps w. OpenTracing
31mLong-Term Metrics with M3 and Prometheus