Monitoring: Turning Noise into Signal
summary
Argues that modest structure and event-style logs turn noisy free-text into actionable monitoring: implement structured events (timestamp, name, value), pass correlation ID through services, and publish local aggregates to avoid massive joins in Splunk or Log Parser. Builds tiny parsers and scripts that auto-detect recurring DB reports and auto-terminate long queries, reducing repeated pager incidents; finds surprise offenders (queries returning millions of rows, disk queue length skyrocketing, 30s logging stalls) and reclassifies noisy 400/500s so dashboards become usable.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:17Chris Oldwood
- 2:17The Great Divide
- 3:30Dev+Ops
- 5:01Multiple Narratives
- 8:53Full Pipeline Developer
- 10:38Self-Healing Systems
- 19:35Instrument Everything
- 30:25Transient Blips
- 31:32Long-Term Trends
- 36:30Free-Text
- 39:50Aggregatable Logging
- 41:49Minimal Event Structure
- 42:57Naming
- 48:33Diagnostic Context
- 51:23Causality
- 55:07Local Aggregations
related talks
59mMonitoring A-Z
18mFrom Logs To Signals: Rethinking System Monitoring With OpenSearch
15mAnti-patterns in Security Monitoring
41mMonitoring Big Data Systems Done "The Simple Way"
19mSensory Friendly Monitoring: Keeping the Noise Down
19mSensory Friendly Monitoring: Keeping the Noise Down
55mBeginner's guide to observability
36mBeyond grep: Practical Logging and Metrics