From Statsd To OpenTelemetry: Atlassian's Metrics Platform Migration…
summary
Atlassian migrates a 10-year StatsD pipeline to OpenTelemetry Collectors, replacing go-statsd sidecars with collector-based sidecars and gateways and keeping zero downtime and no data loss. The pipeline processes about 4.8 billion datapoints per minute and achieves roughly a 95% reduction via stateful per-timeseries aggregation. Key changes: a load balancing exporter using stream ID consistent hashing to eliminate hot aggregator shards, a custom delta-aggregation processor, and lower CPU and smoother scaling in production.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:02Why Atlassian needed metrics
- 3:08Challenges with go-statsd
- 4:36Why migrate to OTel
- 5:39OTel collector architecture
- 9:52Swapping StatsD for OTel
- 10:56Atlassian’s aggregation layer
- 13:24Hot aggregator shards
- 15:12Load balancing with Stream ID
- 17:20Stream ID impact on load
- 19:09OTel aggregation with delta
- 20:51Performance gains from OTel
- 21:55Forwarding stage and exporters
- 23:45Final OTel pipeline
- 24:28Migration lessons
- 26:37Adopting OTel at scale
- 28:04Closing and Q&A
related talks
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
25mAdopting Open Telemetry Collector @ eBay - Swapping Engines Mid Flight
28mThings I wish I knew before we decided to migrate our metrics infrastructure..
30mTips And Tricks To Successfully Migrate From Jaeger To OpenTelemetry
36mPanel: OpenTelemetry...
16mUsing the OpenTelemetry Collector to Empower End Users
34mPanel: OpenTelemetry: Bridging Platform and Enablement