Raise a Production System:Making on-call human-scale w/ Observability & Trust
summary
A team of under ten engineers makes every engineer — even front-end — take full-stack on-call duty, collapses two rotations into one, and relies on Honeycomb instrumentation, traces and a production first-aid kit (notably feature flags) to triage. Deploy-window metadata and traces expose a year-long build slowdown and a single errant host, enabling quick rollbacks or config toggles. Outcome: on-call becomes manageable, knowledge spreads, debugging improves, and incident work shifts toward prioritization and learning rather than blame.
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 3:17What is observability
- 5:30On-call rotations at Honeycomb
- 9:44Democratizing debugging
- 13:20Pairing and knowledge sharing
- 18:00Engineering culture and curiosity
- 20:31Managing on-call anxiety
related talks
44mYou build it, you run it
29mCalling out a terrible on call system
50mYou build it, you run it (why developers should also be on call)
30mOn call does not have to suck
31mHow to Build a Healthy On-call Culture
19mOn-call revolution: Fostering ownership and collaboration
16mHuman Centered On Call Practices
26mBuild a data-driven on-call workflow for your team with atomic habits