The Virtuous Cycle: Getting Good Things out of Bad Failures
summary
Heroku runs over a million Postgres databases, tens of thousands of Redis instances and hundreds of thousands of dynos and the same integer auto‑increment rollover hits production repeatedly — it happens three times before they fix it. They add pg_diagnose alerts at 75% and 90% consumption, add a productionization checklist to enforce bigint sequences, push an ActiveRecord default change, and embed incident tooling in chat (chatops, sitreps, bot-driven timelines) to support retrospectives within a week and produce a long remediation backlog.
AI-written summary · about
chapters (5)
related talks
30mAbove the Clouds: Investigating Systems from 32,000 ft.
42mPlanning To Fail
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
41mRestarts and rollbacks don't fix everything
11mUsing incidents to level-up your teams
30mUsing incidents to level-up your teams
34mThe Doctor Is In: Using checkups to find bugs in production
40mThe Art of Embracing Failure at Scale