What I Wish I Knew before Going On-call
summary
Yelp ad platform teams report that about two thirds of new engineers feel unready for on‑call and it typically takes three to six months to ramp; they build a staged curriculum (one‑day academy, on‑point rotation, two‑hour intro, shadowing) and a searchable runbook template mapping alerts to exact mitigation commands. Concrete failures include a MapReduce analytics job timing out after a schema change that blocks billing via Cassandra; teams run interactive wargames (using the open source twine text‑adventure tool) to rehearse fixes and improve docs.
AI-written summary · about
chapters (59)
- 0:00Intro
- 0:24What is Yelp
- 0:48Why we are here
- 2:16Our story
- 2:33How do we struggle
- 3:33Did you feel ready
- 3:51Why
- 6:20Agenda
- 7:38Common Myths
- 8:30I need to solve everything by myself
- 9:15Find the root cause
- 10:26Stop the bleeding
- 11:44Set the right expectation
- 12:06During training
- 13:48What was good
- 14:33What was helpful
- 15:01What was difficult
- 15:18Deep Dives
- 15:35Investigation Tools
- 16:35Agenda Tools
- 17:09Curriculum Design
- 17:35Avoid Information Overload
- 17:57Uncle Training Program
- 18:35Ad System Overview
- 18:47Billing Pipeline
- 19:10Order of Topics
- 19:55Exercise
- 25:17Overview of the system
- 26:47Example
- 27:49Visual Aid
- 31:01Twitter
- 33:24Failure Modes
- 34:24Data Processing Failure
- 38:23Necessary Tools
- 38:52Service Dashboard
- 39:26Design Your Own Boring Program
- 39:36Signal Effect Dashboard
- 40:02What might have gone wrong
- 40:15List tools and knowhow
- 43:23Congratulations
- 43:54Recap
- 44:15Wynton
- 45:17Neon Color Engineers
- 45:31Oncall Handoff
- 46:19Postmortem
- 47:47Wargame
- 48:07Dungeon Dragon
- 48:17War Game
- 48:42Game Master
- 48:57Player
- 49:44Scenario
- 51:12Preparing the Game
- 53:14How Many Players
- 54:24Hints
- 55:07Invite Everyone
- 55:29Clarify Questions
- 56:07Take Some Note
- 56:37Hardest Part
- 57:03Low Cost Option
related talks
24mWait for Us! Evolving On-Call as Your Company Grows
35m“On-Call Is Ruining My Life” and Other Tales about Holding the Pager as an SRE
13mZero to the world - How to find your feet with the tech when joining a new company
06mEffective Onboarding: From Zero to Full Speed
22mHumane On-call
32mMy Great Awakening: Top “Aha” Moments As Former Dev Manager
50mPanel: Engineering Onboarding
28mTechnical Onboarding, Training, and Mentoring