Navigating in the Dark
summary
Observability, paging and incident tools fail alongside production systems — even with correlated failures from shared infrastructure — illustrated by Air Canada Flight 143 (July 23, 1983) and concrete outages like AWS Kinesis affecting CloudWatch and a Docker awslogs driver bug that buffers and drops logs under load. Built workarounds that prove useful: an internal status page plus an externally monitored lightweight page, a global playbook chapter, no data alerts enabled, dual alerting (email + pager), local cyclic logs or Prometheus endpoints, and game days that simulate losing Slack or the metrics/log store.
AI-written summary · about
chapters (21)
- 0:00Intro
- 3:08Flight Manual
- 3:49Subsystems
- 6:09Aircraft Incident
- 6:54Who am I
- 7:10What can we take from this story
- 7:21Everything fails all the time
- 7:54Systems that fail
- 14:15What should we do
- 15:21No data
- 16:02Status page
- 16:33PagerDuty
- 17:08Use Case
- 17:26Playbook
- 18:36Know your customers
- 19:08Dont have an outage
- 19:25Configure alerts
- 20:10Dual logging
- 21:01Most effective advice
- 22:00Summary
- 22:30Conclusion
related talks
31mOn the Importance of Visibility
58mObserving Applications in Production
39mConnecting the dots: observability in large e-commerce environments
01h 03mWhispers in the Chaos: Monitoring Weak Signals
29mGaining Insights from a Black Box System
45mWhat to do when complex systems fail: A guide to chaos management
36mDude, You Forgot the Feedback: How Your Open Loop Control...
05mKeynote: Observability in the Next Decade, Beyond Reactive Response