Production Improvement Review: Taking a Bite Out of Repair Debt
summary
Azure SRE runs a Production Improvement Review that forces common KPIs across wildly different teams, tracks time-to-detect/engage/fix, and pushes incident fixes into a parallel repair backlog next to feature work. The process surfaces a new class of missed fixes called virtual debt (PacMan debt) and shows monitoring gaps multiply mitigation time by about 10–15×. Durable cadence—daily triage, weekly service reviews, monthly PIR—and automated dashboards tie RCAs to actionable repairs and measure whether fixes actually reduce incidents.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:10What is Production Improvement
- 2:13Production Improvement Review
- 5:05The virtuous cycle
- 7:22The framework
- 12:45Building dashboards
- 17:13Root cause analysis
- 24:50Cadences
- 27:01Insights
related talks
51mShrinking Production Incidents
29mShrinking the Impact of Production Incidents
40mWhen Incidents Fix Themselves: AI SRE in action
54mUnraveling the Differences
25mRethinking measurement for high-performing teams
37mLearning from Incidents at Scale; Actually Doing Cross-Incident Analysis
25mHow to detangle and understand production
34mIncidents are new normal