20 Years on the Rake: Failures, Outages and Lessons Learned
summary
A 20-year retrospective on production mistakes and outages in software development and operations, covering memorable failures, surprises, and lessons learned. The talk focuses on incident causes and practical takeaways for running high-load systems, rather than just scaling or new technologies.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:39Video call incident
- 13:47Five availability zones down
- 19:28Load balancing errors
- 23:42Microservice cascade failure
- 32:33Worst outage
- 37:56Database quorum loss
- 39:21Key lessons and culture
- 40:26Q&A
related talks
49mLessons from a Project with a 2001 Commit
45mToward a Culture Where Everyone Is Responsible for Outages
27m20 Years of SRE: Highs and Lows
55mA Decade of Progress: The Evolution of Production Engineering and Reliability
50mDevOps Project Failures - Lessons Learned and Strategies for Recovery
39mConfessions of a Systems Engineer: Learning from My 20+ Years of Failure
23mLessons from 25 Years in Infra: What’s Changed, What Hasn’t,...
49mTen Years of the DevOps Movement - A Retrospective and Outlook