Production Incident Evaluation as Cost of Quality
summary
Avito runs incident measurement over 18 months across about 40 cross‑functional teams and sees one incident per working day, classifies each by eight concrete fields (impact, reason, trigger, timeline, detection, actions, platform, team) and uses canary rollouts at 5% plus Slack/SMS alerts and monitoring thresholds (99% responses <200ms, alert if >35% >300ms). Financial triage flags tickets over 20 story points; the largest measured loss reaches 130 points.
AI-written summary · about
chapters (14)
- 0:00Incident evaluation as cost of quality
- 1:58Large teams and incident density
- 3:49Classifying incidents by transparency
- 9:02Avoiding incidents and key principles
- 10:04LifeSite review process
- 12:35Eight characteristics for analysis
- 16:17Incident resolution timeline
- 18:29Action items to prevent incidents
- 21:42Measuring customer impact
- 29:16Scaling incident management
- 32:56Benefits for engineers and managers
- 35:07Q&A: Support collaboration
- 40:56Q&A: Alert setup
- 42:26Q&A: Project managers
related talks
37mLearning from Incidents at Scale; Actually Doing Cross-Incident Analysis
19mIncident Archaeology: Extracting Value from Paperwork and Narratives
26mMeasuring the Success of Incident Management
29mShrinking the Impact of Production Incidents
34mIncidents are new normal
19mHow to handle incidents at scale
37mIncident Management Metrics That Matter
37mIncidents are a new normal