Fixing out-of-hours on-call
summary
Intercom creates a volunteer virtual on‑call team of about six to eight people rotating ~six months, moves alarms into a Terraform + GitHub flow pipeline with required runbooks, and sets a simple escalation path to an on‑call engineering leader. Out‑of‑hours shifts run Friday evening to Friday morning, pay a flat fee, and aim to be first responders only. The result is fewer noisy alerts with most months at <10 pages/month and a global diverse team under an explicit volunteer not conscripts policy.
AI-written summary · about
chapters (20)
- 0:00Start
- 7:09Inconsistent quality and experience.
- 8:09Curiously high tolerance for out-of-hours pages.
- 9:08Oncall was eating our engineers!
- 10:16Virtual team of volunteers.
- 11:09We wrote standards and invented new processes.
- 12:13We moved alarms over to the new virtual team!
- 13:37Outside of office hours, the oncall team gets all pages.
- 13:53Product teams do oncall for their stuff when in the office.
- 14:416 or 7 engineers, 6 months on the team.
- 15:38Escalation via an on call engineering leader.
- 16:40They can then escalate to the engineering team manager.
- 18:26"By the book" escalations were uncommon.
- 20:39paging alarms destroyed.
- 21:22Rotating people in and out of the team actually worked!
- 24:23Monthly communications about how things were going.
- 25:05Membership of the team constantly referred to in anniversaries and performance reviews.
- 25:42Accidentally built a global, diverse, engineer-led virtual team.
- 27:30Challenge every part of your oncall setup.
- 29:43Don't let oncall eat your engineers.
related talks
19mOn-call revolution: Fostering ownership and collaboration
16mHuman Centered On Call Practices
30mOn call does not have to suck
06mIgnite: Key Factors for Successful 24×7 On-Call Duty
29mCalling out a terrible on call system
31mHow to Build a Healthy On-call Culture
23mFirefighting: The Engineer's Guide to On-Call
19mIts Time We Rethink On-Call Culture