Fixing On-Call When Nobody Thinks It's (Too) Broken
summary
Hudson River Trading reduces paging from ~200/week (~70k over 7 years) to under 50/week in months by data-driven fixes: import PagerDuty history into Elasticsearch, visualize hourly spikes, and identify cron-driven timeouts. The team templates Nagios config (15k lines replaced by ~400 lines of Python templates), automates CI deploys, extends a Python alert-sender to silence predictable noise, and creates a Slack channel for silenced alerts to ease anxiety. Result: sustained ~4x reduction and safer, auditable silencing.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:52A Brief History
- 3:59Trade Systems Overview
- 7:009 Really Hard Steps to Reduce Pager Noise
- 7:40Understand Your Audience
- 10:31Understand the Problem
- 12:26Understand the System
- 14:08Devise a Game Plan
- 16:07Get Permission Optional
- 19:39Lay the Groundwork
- 21:01Fix the Lowest Hanging Fruit
- 23:18Communicate, Communicate, Communicate
- 25:27GOTO 7
- 25:56Ten Second Review
- 26:49Sound Fun? We're Hiring
related talks
23mFirefighting: The Engineer's Guide to On-Call
28mJust Enough Reliability
33mHow we went from Slipping to Shipping products
28mTop 5 Things You Can Do To Reduce Operational Load
26mHere's your Pager GLHF
29mCalling out a terrible on call system
34mBuilding a Culture of Reliability
22mBlaming in a Blameless World