How Snowflake Migrated All Alerts and Dashboards to a...
summary
Migrated all alerts and dashboards to a Prometheus-based metrics system in three months, rewriting ~1,200 unique alerts and 200 dashboards across 70+ teams (down from ~1,600 candidates), while keeping the old provider as a fallback. Handles prior scale of ~2M points/sec and ~400M active series, replaces 100k legacy evaluation rules, uses Prometheus-compatible APIs, Jsonnet for configuration-as-code, an internal CLI ("oi") and a unit testing framework (promtool-style tests) with Grafana integration — migration succeeds and uncovers renewed metric growth.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:00Metrics at Snowflake scale
- 5:55Motivations for migration
- 9:26Legacy system trade-offs
- 13:01Prometheus-based system choice
- 16:52Dashboards and alerts as code
- 21:33Updating alerting framework
- 23:20Adapting to PromQL challenges
- 25:25Validating alerts with tests
- 28:53Custom interfaces for reliability
- 30:20Migration process
- 31:22Migration results
related talks
11mBuilding lightweight observability system for Lean Teams
29mMaking tech boring to keep data exciting
42mEvolution of Observability Tools at Pinterest
21mJourney from Broken Dashboards to O11y Maturity
29mApplying GitOps principles for central alert management
06mTying a metrics to every release
23mHow We Built a Complex SLO App Tightly Integrated with Grafana
33mAutomating Data Analytics: Collect, Transform and Deploy - Lead With Change