A Tale of Two Rotations Building a Humane & Effective On Call
Talkat SREcon19 Europe/Middle East/Africa · on Wednesday, 2019-10-02· at 11:00· Track 2: The Liffey A·
summary
Compares two rotations: a core ring zero rotation that reliably mitigates high-impact failures in ~30 minutes and has a training waitlist, versus a vendor-integrations rotation that regularly yields 8-hour firefights because overlapping vendor SLAs (each ~44 minutes downtime) multiply outage time. Builds mitigation tooling (scripting of runbooks, sobriety checks requiring a random four‑digit confirmation, and full audit logs), measures cost-per-minute and incidents/week, and shows that deliberate safeguards plus logging restore trust, increase tool use, and turn frantic escalations into low‑stress stabilization.
AI-written summary · about
chapters (9)
- 0:00On-call rotations at Uber
- 1:10On-call as first response
- 3:14Ring Zero vs vendor integrations
- 7:53Cost and benefits of on-call
- 10:22Mitigation tooling for incidents
- 14:54Safe tooling for engineers
- 18:00Follow-the-sun on-call limits
- 24:50Testing mitigation tools
- 25:58Model for leaderless teams
related talks
19mOn-call revolution: Fostering ownership and collaboration
31mHow to Build a Healthy On-call Culture
29mCalling out a terrible on call system
16mHuman Centered On Call Practices
44mYou build it, you run it
22mRaise a Production System:Making on-call human-scale w/ Observability & Trust
26mBuild a data-driven on-call workflow for your team with atomic habits
30mIncident Groundhog Day