The Many Ways Your Monitoring Is Lying to You
summary
Monitoring systems misrepresent reality: missing points produce fake spikes when deltas are computed, sampling granularity hides short bursts, and automatic downsampling after three hours changes curve shape even though the system behavior stays constant. Aggregation shortcuts use stale values so mean memory looks higher when rapid restarts leave the monitor seeing only ~75 tasks. Histograms with wide buckets can hugely overstate tails — e.g. a true 95th of ~130 ms can be reported as 95th percentile inflated 130→250 ms. Other concrete failures include a 20 MB truncation of exporter payloads and client-side vs server-side discrepancy that hides post-shutdown errors.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:40Overview
- 3:30Example
- 4:05Map is not the territory
- 5:32Timeseries based monitoring
- 7:05Lies of a mission
- 8:32Monitoring example
- 11:55Granularity
- 14:58Graphs
- 15:27Perspective
- 15:47Simple Example
- 16:47ClientSide Example
- 19:31Dashboard Example
- 22:10Lies of Presentation
- 28:35Calculating percentiles
- 32:16Conclusion
related talks
37mLies, Damn Lies, and Metrics
18mLook at Your Data
27mLatency SLOs Done Right
12mModel Monitoring: Detecting and Analyzing Data Issues
37mMonitoring the Health of Your App
47mMonitoring at Scale: Intuitive Dashboard Design
31mWe Monitor Everything… Except What Matters
30mThe subtle art of misleading with Statistics