Replacing four monitoring systems in four years and surviving
summary
Describes replacing four different monitoring systems over four years for a high-load production environment, naming the cause-and-effect failures (five key incidents) that drove each change — including a night-time outage that alerted dozens. Covers practical lessons from those migrations and the operational problems encountered when running monitoring at scale.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 5:48Selection criteria
- 7:57Why change monitoring
- 11:30First switch: Zabbix
- 15:22InfluxDB v1 issues
- 16:45Graphite with ClickHouse
- 20:36Adding Prometheus
- 24:04Switching to Thanos
- 29:26VictoriaMetrics test
- 31:12Summary and lessons
- 34:21Q&A
related talks
39mThe Story of One Monitoring
28mMonitoring Evolution at Scale
20mUsing Monitoring to Speed Up Your DevOps Journey
43mGetting Your System into Production and keeping It There
15mWhy I Prefer Good Testing Over Excellent Testing
58mObserving Applications in Production
33mMonitoring Systems as a Service – Walking the Line between...
06mMonitoring as a First Step to a New Service