Joining The Dots: How Span Links And Fake Traces...
summary
Engineers build supertraces to represent 12‑day deployments as a single logical trace by emitting a synthetic root span (created via a risky reflection hack in .NET Activity) and linking high‑volume, sampled event traces with span links. Pipeline uses RabbitMQ, SQL Server and SMB mounts; OpenTelemetry Collector applies ~1:1000 sampling for events while keeping all deployment traces. A 10‑second forced emit marks stuck work. Root cause for long gaps is slow network file I/O; switching to HTTP file ops and payload-on-disk reduces pauses and validates the approach.
AI-written summary · about
chapters (15)
- 0:00Introduction and Kanizsa triangle
- 1:03OTel limitations for long deployments
- 2:46Challenges with automatic instrumentation
- 4:10Deployment as a trace structure
- 5:57Root span memory constraints
- 7:01Hacking .NET Activity API
- 8:44Backwards emission of deployment traces
- 9:26Span links for event traces
- 10:10Deployment duration surprises
- 11:12SQL Server and RabbitMQ choice
- 12:37Diagnosing file system bottlenecks
- 14:00Halting problem in deployments
- 16:32Timers for stuck deployments
- 17:36Future work and randomness value
- 18:40Q&A and acknowledgments
