A Decade of Progress: The Evolution of Production Engineering and Reliability
summary
History of Production Engineering at Meta: how appops and SRE split into production engineering and site reliable operations, concrete automation wins (mcash warm‑replication to avoid cold cache rebuilds, container manager that shrinks ops headcount), observability systems like ODS and Scuba in‑memory analytics, and networking primitives like service router / Thrift. Operational lessons: reduced metrics flush from 5m to 1m (detection 15m→3m), emergency tactic DNS TTL 30s→5s then blackhole, use blameless incident reviews, feature flags and measured release cadence to trade safety for speed.
AI-written summary · about
chapters (11)
- 0:00Introductions and backgrounds
- 2:24Early career and motivation
- 9:20Creating the PE discipline
- 13:07Scaling and managing complexity
- 17:17Balancing change and stability
- 22:25Leadership and cultural shifts
- 26:09The future of the discipline
- 31:05Hiring and team dynamics
- 37:58Incident management stories
- 46:51Impacting systems
- 50:58Advice for building PE teams
related talks
28mReliability Engineering: What we learnt so far
01h 39mPostOps: A Non-Surgical Tale of Software, Fragility, and Reliability
43mFrom "Keeping the Lights On" to "Designing the LEDs": A Detailed Review...
51mThe Evolution of Site Reliability Engineering
58mDevOps @ Scale — Greek Tragedy in Three Acts
28mDevOps in the Enterprise: A Transformation Journey
20mDoing Things the Hard Way
29mRed Hat IT’s DevOps Journey: Year One Retrospective