On the Importance of Visibility
summary
A team builds a composable monitoring tool of AWS Lambda functions that queries CloudWatch, checks known internal IPs and drops findings into Slack; messages stop arriving for about six weeks because a Slack webhook disabled (it was owned by a user who was removed). Fixing it with a headless user and updating the function restores alerts in days. They adopt an incident management framework (inspired by Etsy/Dropbox/Netflix), endure three weeks of chaotic onboarding, then reduce recurrence and centralize postmortems.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:52Overview
- 1:45Visibility
- 4:23Visibility is an Onion
- 5:12Our Visibility Dilemma
- 6:30Our Solution
- 12:02Integrations
- 12:39The Problem
- 17:33Incidents
- 18:08Incident Management Framework
- 24:10Mayhem
- 26:12Takeaways
- 29:39Making Things Visible
- 30:15Wrap Up
related talks
31mPreparing for downtime when all you want is uptime
49mMonitoring serverless applications on AWS: the monitoring monster becomes your friend.
26mHow to survive a Lambda-induced data tsunami in AWS
57mThe Curious Incident in your Software in the Day Time
23mNavigating in the Dark
29mWhat could possibly go wrong?
28mThe Agent That Turned On-Call Chaos into a Single Slack Thread
42mCloudWatch-ing: Creating More Useful Logs & Alerts with AWS