Learning at Scale Is Hard! Outage Pattern Analysis and Dirty Data
summary
Microsoft Azure teams build a pipeline that starts with post-mortem text clustering (cleaning HTML with BeautifulSoup), fits ML models but finds model fitting is trivial compared with data work, and ranks outage themes with a pain-value metric (count × duration × severity × weighting like customers/regions/SLO breaches). Dirty data—missing postmortems, ambiguity, config drift, typos—forces manual cleaning and confidence labels (signal bars). Outcome: several high‑pain issues surface that were absent from backlogs, many myths get debunked, and cross‑correlating themes with metrics reveals minor incidents that predict major outages.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:25Learning (From Failure) At Scale
- 1:53Problem Management
- 2:58Gathering data
- 3:36Fundamental Issue: ROOT CAUSES
- 4:03Complex systems fail in complex ways
- 5:07To stop bad stuff from happening (again)
- 5:25Hunting for Causes Problems Contributing Factors
- 6:01Postmortem Text Analysis
- 10:53Ambiguity
- 11:48Dirty data will lie to you.
- 12:23Surfaced surprise issues
- 12:54Debunked production myths
- 13:23Stronger arguments for prioritization of reliability work
- 13:46Define your hypotheses
- 14:03Clean your data
- 14:25Cross-Correlate Data Sets
- 14:46Study your minor failures
- 15:14Intelligently calculate Risk
- 15:39Continue to improve the RCA Process
related talks
30mBetter data quality with ML & visualization
32mReliable Machine Learning
34mData Science beasts (failures) and where to find them
28mHindsight bias: A machine learning story
49mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
25mAutomating Operations with ML
11mTransforming Problem-Solving with Data-Driven Insights
19mHow ML Breaks: A Decade of Outages for One Large ML Pipeline