"What Happened in Production?! Instrumenting with OpenTelemetry"
summary
Daily 2pm crashes turn out not to be a mystery but an OpenTelemetry Tracing story: Python auto‑instrumentation reveals DB deadlocks that metrics and logs miss, manual spans add business context, and sampling controls cost — e.g. head-based sampling (one in 20) vs tail-based sampling to keep all error traces. A zero‑code bootstrap gets spans flowing to Honeycomb, a staged demo of BrisBoy attempts to reproduce the failure but the live incident never appears so the instrumentation itself is validated but the crash is not reproduced.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:50AI slop to observability
- 5:46Introducing Ollie
- 7:09Three pillars debunked
- 9:58Logs fall short
- 13:09Tracing limits
- 14:34What is OpenTelemetry and why it matters
- 17:07Tracing under the hood
- 19:14Practical OpenTelemetry
- 20:59Tracing sampling
- 22:23Getting started
- 24:50Live demo
- 27:17Summary
related talks
31mTroubleshooting and Monitoring Production Applications using OpenTelemetry
22mOpen Telemetry for API Monitoring
02h 02mHow To Troubleshoot and Monitor Applications using OpenTelemetry
33mThe Observant Developer - OpenTelemetry from code to prod
01h 28mUnderstanding Distributed Systems with OpenTelemetry
42mObservable Applications Using OpenTelemetry
26mWelcome to the OTEL California
50mBootiful OpenTelemetry