Observing from Incidents
summary
Companies mine Slack incident transcripts with Jelly to extract the concrete questions people ask and then encode those signals into dashboards: link runbooks, show SLO targets and units, overlay alert events and deploys on charts, lazy‑load heavy GC panels, and display automation actions as annotated triangles so humans and bots share context. Stripe uses canonical log lines to precompute “who was affected” queries. These changes reduce mean time to remediation and cut confusing dead ends; they are pragmatic, incremental, and measurably useful.
AI-written summary · about
chapters (10)
- 0:00Introduction and philosophy
- 2:12Rethinking incident learning
- 7:18Dashboards as user interfaces
- 10:34Analyzing incident logs
- 13:44Dashboard hygiene techniques
- 19:16Context and clarity
- 23:06Boundaries and signaling
- 29:48Automation and context
- 34:28Situational awareness
- 38:40Conclusion and resources
related talks
23mHow Tracing Uncovers Half-truths in Slack’s CI Infrastructure
34mIncidents Happen
27mLet’s Talk About Time - Data Driven Organic Monitoring
12mQ & A
42mMachine Learning In Observability Systems: From Insights To Action
19mImproving Reliability @ Meta: By Analyzing Historical Events That Led to SLO Violations
57mThe Curious Incident in your Software in the Day Time
37mLearning from Incidents at Scale; Actually Doing Cross-Incident Analysis