Familiar Smells I’ve Detected in Your Systems Engineering Organization and How to Fix Them
summary
Groups get paged 90 times in six hours because alerts are noisy, runbooks are untested Word docs and short‑term manual fixes (like outsourced ops deleting logs) persist for years. The claim: stop remediating the same failure and fix causes — use the PagerDuty API to list incidents, add unit‑tested infrastructure code, use representative stages like prod, track SLOs and MTTR, run gamedays and chaos experiments, and prefer simple, automatable tech that you can reliably recover.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:36Why am I calling them smells
- 5:58Outsourcing
- 8:03Stockholm Syndrome
- 9:24Fix Problems
- 10:40PagerDuty
- 11:38Structure
- 12:48Alignment
- 14:18Untested Infrastructure
- 14:38Run Books
- 16:17Review Changes
- 16:55Test Infrastructure
- 18:58Continuous Delivery
- 19:48Boring Technology
- 20:51Not Having Failure
- 22:36Black Swans
- 23:14Netflix
- 23:47The Next Internet
- 25:10Production Readiness
- 26:01Chaos Engineering
- 26:42Things to Take Away
- 26:52Questions
related talks
45mHow to Infect your Organization with Humane Ops
45mIncident Management - Talk the Talk, Walk the Walk
30mReliability Engineering for Humans
28mTop 5 Things You Can Do To Reduce Operational Load
26mMission Critical Mindset Building a Culture of Resilience in DevOps Teams
32mIncident Management - Talk the Talk, Walk the Walk
19mYou Can't Stop Fires with an Ambulance
01h 04mIncident Management - Talk the Talk, Walk the Walk