Lessons From Long-Running Investigations
summary
Describes concrete investigative practices: using divide and conquer by running parallel experiments across a fleet to spot divergence and pinpoint a regression to the version that diverged (example: versions 98 → 100); discovering a hidden years-old regression when a fix drives metrics well below the accepted baseline, i e baseline drift; enforcing an audit trail and communication when teams must break the glass for live experiments; and upgrading observability to one second or subsecond sampling and automated high-frequency telemetry to capture the true cause‑then‑effect.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:50Talk expectations
- 5:19Divide and conquer debugging
- 10:56Audit trail during incidents
- 13:46Breaking glass investigations
- 15:14Resolution in telemetry
- 19:47Resolution and root causes
related talks
37mBetter Reliability Through Observability and Experimentation
39mHow to Use Common Investigation Practices in Troubleshooting
39mResolving Outages Faster with Better Debugging Strategies
41mA peek into observability from testers lens
25mHow to detangle and understand production
30mHow Did Things Go Right? Learning More from Incidents
29mGaining Insights from a Black Box System
38mOff the Beaten Path: Moving Observability Focus from Your Service