Learning from Learnings: Anatomy of Three Incidents
summary
An App Engine 8-hour global outage in October 2012 triggers a timeboxed, whiteboard-driven program: teams bucket eight-to-ten themes, spend one week scoping projects, track work in a simple spreadsheet, and increase router capacity; the year-long effort yields a 10x reduction in reliability incidents. At Stitch Fix, >50% of outages trace to a shared database so teams cut logging, rewrite hot queries, use a connection concentrator and commit ~25% engineering investment to migrate services off the shared DB. WeWork faces inconsistent microservice views causing login failures and is mid-implementation on the same pattern.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:40Background
- 1:07App Engine Outage Oct 2012
- 3:03App Engine Reliability Fixit
- 9:14Database Stability Problems
- 9:53Stitch Fix Stability Retrospective
- 10:38Stability Solutions
- 11:21WeWork
- 13:15Common Elements
- 15:57Blameless Post-Mortems
- 17:30Cross-Functional Collaboration
- 21:55Definition of Done
- 22:21of Technical Debt
- 22:55of Investment
- 23:17Lessons
- 25:16Funding the Improvement Program
- 29:17We are Hiring!
related talks
48mMoving Fast at Scale
16mCreating a Learning Culture
28mAfter the dust has settled: learning from incidents
18mMaking what you have enough
28mThe Failing CTO - What we can learn from others' mistakes
29mApps, Stacks, and Frameworks: Avoiding “Shiny Object” Syndrome
20mDoing Things the Hard Way
41mYour Systems Aren't Stuck - Your Teams Are