The Sounds of Silence: Lessons from an 18 hour API outage
summary
An 18‑hour outage fixes concrete lessons: a backfill corrupts records so nullified foreign keys in two screening tables cause ActiveRecord::RecordNotFound and return 404s for some report creations, a statsD counter and a hard threshold of 100 reports in 30 minutes miss partial failures, VPN collapse slows on‑call access, a temporary patch takes 15 minutes but time to response gaps (about 14 hours) multiply impact. Follow ups add Sentry/APM/custom metrics and a composite monitor tying report and screening creation; Datadog anomaly detection proves noisy.
AI-written summary · about
chapters (12)
- 0:00Introduction to the outage
- 1:32Context and report system
- 2:56Incident timeline and detection
- 5:00Resolution and investigation
- 6:03Post-mortem and root causes
- 9:21Analyzing delayed detection
- 12:25Observability pillars
- 13:53Types of metric collection
- 17:37Crafting actionable alerts
- 19:24Monitor patterns and tools
- 21:59Building an observability culture
- 25:01Conclusion and takeaways
related talks
23mSquash Production Defects Quickly - The Power of Structured Logging in Rails
34m"The Upside of Downtime: How to Turn a Disaster Into an Opportunity"
44mBlack Monday: The Story of 5.5 Hours of Downtime
25m"The website's down!" Stories and lessons on keeping your website up.
30mIt's Down! Simulating Incidents in Production
52mPerformance & Stability Anti patterns
23mDiving through RCAs a quest for reliability
23mNavigating in the Dark