Disintegrated telemetry: The pains of monitoring asynchronous workflows
summary
Asynchronous, temporally decoupled workflows create gaps that tear traces into chunks, so simple root-span tracing breaks at scale. The practical solution uses either a single long trace or multiple traces correlated with producer span and consumer span connected by a span link; OpenTelemetry messaging semantic conventions prescribe that invariant. W3C drafts for AMQP/MQTT context propagation exist but are unstable, sampling for linked traces is unresolved, vendor support is uneven though improving, and few instrumentation libraries implement the new conventions.
AI-written summary · about
chapters (11)
- 0:00Introduction and motivation
- 1:03Defining asynchronous workflows
- 3:15Example of asynchronous workflow
- 5:24Temporal decoupling and gaps
- 7:56Modeling workflows as traces
- 9:41Span relationships and links
- 14:01Pain points in monitoring
- 16:15OpenTelemetry semantic conventions
- 22:22Current limitations and future work
- 24:31Roadmap and calls for help
- 27:40Q&A session
related talks
26mThe Art of Event Driven Observability with OpenTelemetry
22mThe Life of a Span
23mEnhancing Asynchronous Communication Observability with OpenTelemetry
38mPractical introduction to OpenTelemetry tracing
49mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
41mTracing Real-Time Distributed Systems
34mReduce Complexity with Distributing Tracing
54mA TRACE TO REMEMBER: COMPOSITIONAL TRACING