Using incidents to level-up your teams
summary
An API outage traces to a Postgres query plan regression: teams disable a bad endpoint, make a subtle SQL change to restore the inexpensive plan, and then add statement timeouts to limit blast radius. A CSV batch of ~1,000 meters fails because Excel coerces a serial with an “e” into scientific notation; adding an apostrophe workaround fixes uploads. A partner breaks on lowercased headers after a component upgrade and rolling back restores compatibility. Recommendation: use public incident channels, log commands and theories, and discourage lone heroes.
AI-written summary · about
chapters (7)
- 0:00Start
- 3:41Incidents broaden your horizons
- 5:48Incidents teach you to build things that fail gracefully
- 7:00Incidents are a chance for blue sky thinking
- 13:26Incidents teach you to make systems easier to debug
- 16:41Incidents build your network
- 18:00Incidents are a chance to learn from the best
related talks
11mUsing incidents to level-up your teams
20mData Incidents - When things go wrong
37mIncidents are a new normal
19mHow to handle incidents at scale
30mIncident Groundhog Day
57mThe Curious Incident in your Software in the Day Time
30mThe Meat of It
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures