How to Infect your Organization with Humane Ops
summary
A survey across 10,000 companies with 50,000 responders finds 760 million notifications total and 250 million during sleeping hours, and shows frequent weekend or overnight interrupts drive turnover; PagerDuty’s operational playbook recommends blameless postmortems, incident commander rotations and monitoring parity between preprod and prod. Concrete actions: review alerts daily, cap noisy monitors, make metrics tie to business outcomes not punishment, define roles on incident bridges, rally fast and disband faster, and deliver one small responder-focused improvement each sprint.
AI-written summary · about
chapters (25)
- 0:00Start
- 0:57Why Do We Care about this Idea of Making Ops Humane
- 5:15Context Switching
- 8:55Ideology Is a Virus
- 10:10Use Measurement for Good Not for Evil
- 10:58Avoid the Executive Swoop
- 12:04Blameless Post Mortem
- 12:53A Culture of Learning
- 14:23Use Your Smart People
- 14:59Broken Window Effect
- 17:10Cost To Replace a Software Engineer
- 21:21Normalization of Deviance
- 23:40Questioning Metrics
- 30:29Talk to People
- 31:27Who Are Your Customers
- 32:57What Are the Perceptions of Your Team
- 35:25And We Explicitly Allow Anybody in the Organization To Be an Incident Commander whether It's a Product Owner Can Be Somebody in Marketing It Could Be an Engineer It Can Be an Evangelist Even So this Is Something You Could Adopt in Your Organization and It Can Really Make Things Better because You'Re Now Part of this so You Might Be Sitting There and Saying You Know What I Don't I Don't Know How To Solve these Problems That Happen after Hours but I'D Like To Be Able To Help Working into some Kind of Incident Command Can with that
- 36:32Go Look at the Logs on the Web Server I'M GonNa Check with You in Five Minutes and See if You'Ve Seen Anything Interesting There We Need To Be Explicit and Then Rally Fast Disband Faster so It Is Not Pleasant To Be on an after-Hours Call Nobody Wants To Be on that Call We Need To Get the Right People In There As Quickly as Possible but Even More Important Need To Get Them off the Phone As Soon as You Have What You Need Don't Worry You Can Get Them Back if You Need Them It's Incredibly Frustrating To Be Sitting on a Bridge with Nothing To Do
- 36:45So It Is Not Pleasant To Be on an after-Hours Call Nobody Wants To Be on that Call We Need To Get the Right People In There As Quickly as Possible but Even More Important Need To Get Them off the Phone As Soon as You Have What You Need Don't Worry You Can Get Them Back if You Need Them It's Incredibly Frustrating To Be Sitting on a Bridge with Nothing To Do while You'Re Waiting for Someone Else To Do Work so Anyone Ever Sat on a Bridge of a Hundred or More People How Effective Did You Feel during that Call Most of the Time on a Scale of One to Ten It's Probably about a Negative Two and It's Frustrating for the People That Are Doing the Work because They Know There's a Bunch of People Sitting There Getting Irritated
- 37:51The Last Thing You Want To Be Doing When You'Re Trying To Solve a Problem or You'Re Trying To Have People Help You Solve a Problem Is Get into an Argument about How Important It Is for that To Be Being Done Right Now You Don't Want that Escalation To Occur You Don't Want It To Be like Well Well We Decided You Know cuz Sometimes You Have a Best Effort Sometimes It Might Be on a Severity Three Maybe We'Re GonNa Get on the Bridge and We'Re GonNa Try To Fix It for an Hour and Then We'Re GonNa Roll Back We Need To Already Know that We Don't Want To Be Trying To Figure It Out while It's Happening so We Said Don't Litigate during the Call
- 39:21All Functional Tests That's Used in Pre-Production Should Have Parity Should Have an Equivalent Monitor in Production if You Care if this Is Your Way of Knowing that Stuff Works You You Really Want To Make Sure It's Still Working in Production because Who Cares if It Worked in Pre Prod if It Isn't Working in Prod and Likewise if I'M Monitoring Something in Production There Should Be an Equivalent Test in Pre Prod and the Way To Think about this Is that Monitoring Is Nothing but Testing with a Time Dimension That's all We Do When We Monitor Now Again We Can Start Talking about Observability and Starting To Understand Troubleshooting We'Re Not Talking about that yet We'Re Just Talking about Pure Monitoring
- 39:42If I'M Monitoring Something in Production There Should Be an Equivalent Test in Pre Prod and the Way To Think about this Is that Monitoring Is Nothing but Testing with a Time Dimension That's all We Do When We Monitor Now Again We Can Start Talking about Observability and Starting To Understand Troubleshooting We'Re Not Talking about that yet We'Re Just Talking about Pure Monitoring It's Testing with the Time Dimension There Should Be Full Parity between Pre-Production and Production Notice That I Said Functional
- 39:58We'Re Not Talking about that yet We'Re Just Talking about Pure Monitoring It's Testing with the Time Dimension There Should Be Full Parity between Pre-Production and Production Notice That I Said Functional and We Can Get into a Fight about What's a Smoke Test What's a Functional Test but Things like Unit Tests Yes I'M Not Going To Put that into Nagios I'M Not Testing a Class but Is a System Work Yes or no Right Is It Performant Yes or No if I Care about It Enough To Decide that this Says It Can Be Released
- 42:51Put some Context into the Configuration That Says this Is What this Does More Importantly this Is How To Turn It Off and When You Should Turn It Off Um if You Use Run Books Make Sure They'Re up to Date every Time You Cut a Release if You'Re Not Able To Do this Get Rid of the Run Book Altogether an Incorrect Run Book Is Considered Harmful So if You Can't Do this Don't Have Run Books You Might Be Getting Where I'M Going at Which Is Don't Have Run Books because It's Really Really Hard To Keep Them up to Date and Then Simplify Man Again Find Something within Your Code within Your Project that You Can Do a Little Thing To Make It Less Complex
- 43:15You Might Be Getting Where I'M Going at Which Is Don't Have Run Books because It's Really Really Hard To Keep Them up to Date and Then Simplify Man Again Find Something within Your Code within Your Project that You Can Do a Little Thing To Make It Less Complex So I Love for Later for You Guys for You People You Folks Y'all To Share some of Your on-Call Stories with Me I Will Be around all Day Today I'Ll Be Around Tomorrow
related talks
33mFamiliar Smells I’ve Detected in Your Systems Engineering Organization and How to Fix Them
28mTop 5 Things You Can Do To Reduce Operational Load
49mData Driven Observability & Incident Response
16mHuman Centered On Call Practices
23mFirefighting: The Engineer's Guide to On-Call
45mIncident Management - Talk the Talk, Walk the Walk
19mSensory Friendly Monitoring: Keeping the Noise Down
27mSensory Friendly Monitoring: Keeping the Noise Down