Confessions of a Systems Engineer: Learning from My 20+ Years of Failure
summary
No change is safe: a config meant for test once reaches prod via a VIP and takes out prod services, so every rollout uses canary under deployment with staged ramp and positive validation. Automation must measure, diagnose and mitigate — systems fail faster than humans. Lack of admin tooling forces a full index rebuild that costs hours; exercise DR, verify backups, and treat operational tools as production code. Design degraded modes, aggressive traffic redirecting, input sanitization, and documented handoffs.
AI-written summary · about
chapters (15)
- 0:00Intro
- 2:51There Are No Safe Changes
- 4:12Minimize the Blast Radius on Changes
- 6:35Monitor Accurately and Measure Thoroughly
- 9:07Automate Mitigations
- 12:03Degraded Service Modes, or An Imperfect Experience Usually Beats a Nonexistent One
- 15:17Use Functional Gates Pre-, Post- and During Releases
- 18:36Design to Meet SLAs and Mitigate Incidents Quickly
- 21:02Regularly Exercise All Processes and Tools
- 24:15Enforce Processes with Technology
- 27:10Redirect or Drop Traffic Aggressively During Incidents
- 29:01Production Quality Tools
- 31:14Sanitize and Verify Inputs
- 33:17Understand All of the Scenarios You Support
- 35:45Transition Service Responsibilities Carefully
related talks
48mHarnessing Chaos; the hidden ingredient behind building better systems
31mHow to be a Failure
54mRuin Your Systems By Making Them Perfect
31m"Failure" as Success: The Mindset, the Methods, and the Landmines
29mThe Curious SRE: Cultivating a Mindset for System Resilience
25mSurviving DevOps
22mHow to be an inefficient developer?
42mHow to foster a Culture of Resilience