The Math behind the Incident Aftermath: A Practical Guide to Measuring...
summary
PayPal builds an automated Incident Impact Calculation Framework that pulls per‑minute payment counts from observability APIs, computes a six‑week trimmed average (dropping max/min outliers) as the expected baseline, subtracts actuals minute‑by‑minute, and discounts spikes using a recapture adjustment to get net transaction and revenue loss. Architecture is a web UI plus an impact calculation service that writes to ticketing; ML (ARIMA, LightGBM) was tested but the trimmed‑average proved faster and more practical at global scale (5.6B transactions, $337B TPV in Q3 2022).
AI-written summary · about
chapters (11)
- 0:00Incident impact measurement
- 3:35Incident lifecycle challenges
- 7:07Manual impact limitations
- 9:57Core business metrics
- 13:28Tool design overview
- 17:45Six-week trimmed average
- 23:46Recapture calculation
- 25:09ML model trials
- 28:41Additional impact metrics
- 30:49Framework for other units
- 33:39Adopting the framework
related talks
44mProduction Incident Evaluation as Cost of Quality
20mCertainty Among the Chaos
32mKnow your data the stats behind the Alters
30mBeyond the Dashboard: Build End-to-End O11y
19mIncident Archaeology: Extracting Value from Paperwork and Narratives
33mThe Problem with MTTR: Learning from Incident Reports
29mCulturing Resiliency with Data: A Taxonomy of Outages
28mBeyond the Balance Sheet: Transactional Data as a Window to Company Health