Using incidents to level-up your teams
summary
API slows to a crawl because one endpoint times out; team disabled endpoint to restore service, then diagnoses a Postgres query plan regression and deploys a query change that reverts the planner choice. Follow-ups include adding statement timeouts (2–3s) to limit blast radius, publishing incident channels and debriefs for onboarding, lowering the bar to declare incidents, and avoiding a single “hero” who hoards learning and burns out.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:58Incidents broaden your horizons
- 3:58Incidents teach you to build things that fail gracefully
- 4:49Incidents teach you to make systems easier to debug
- 5:31Incidents are a chance to learn from the best
- 7:13How do we make incidents accessible?
related talks
30mUsing incidents to level-up your teams
19mHow to handle incidents at scale
30mIncident Groundhog Day
37mIncidents are a new normal
20mData Incidents - When things go wrong
27mManaging Incidents
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures
31mThe Art of Turning Incidents into Opportunities