Things I wish I knew before we decided to migrate our metrics infrastructure..
summary
Migration moves seven things at once: instrumentation from statsD to OpenTelemetry, collection (OTel collector sidecars with a patched statsd receiver), a custom aggregation tier with consistent hashing sharding, storage tuned for bursts and multicluster federation, Grafana for UI, PromQL training, and a new alerting pipeline. Surprises: tens of thousands of dashboards and hundreds of thousands of alerts to translate, 30s aggressive query timeouts, and the need to rewrite aggregation and scale the storage layer operationally.
AI-written summary · about
chapters (15)
- 0:00Introduction and motivation
- 3:34Why migrate in-house
- 6:01Seven migrations in one
- 7:06Instrumentation challenges
- 10:18Collection tier migration
- 11:42Aggregation pipeline changes
- 13:07Storage cluster scaling
- 16:03Multi-cluster setup
- 17:47Circular dependency issues
- 19:17UI and alerting migration
- 20:43Cultural and team changes
- 21:45Migration strategies
- 23:31Minimizing vendor lock-in
- 25:59Key takeaways and risks
- 27:02Q&A: Cost-based migration timing
related talks
29mFrom Statsd To OpenTelemetry: Atlassian's Metrics Platform Migration…
52mBuilding a culture of observability
31mCloud Migration Without Fear - A...
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
28mFrom Legacy Vendor Tooling To OTEL: Scaling Observability at MSCI With a Four-Person Team
35mDoorDash’s Journey From StatsD To Prometheus With 10 Million Metrics/Second
28mMonitoring Stack was yesterday – Observability Platform at scale!
44mGrafana Labs Customers: What We've Learned Building Observability at Massive Scale