DoorDash’s Journey From StatsD To Prometheus With 10 Million Metrics/Second
Talkat KubeCon + CloudNativeCon North America 2022 · on Friday, 2022-10-28· at 11:00· Portside Ballroom (Room 260)·
summary
DoorDash migrates its monitoring from StatsD to Prometheus, moving from delta counters and precomputed percentiles to histograms not percentiles, deploying Prometheus collectors as per-node agents and using Prometheus client libraries and exporters (jvm_exporter, statsd shims, yc e e exporter) with Kubernetes annotations and Terraform-managed dashboards and alerts. The pipeline ingests ~15M metrics/sec and persists ~10 million metrics per second after aggregation, uses relabeling, rollup and mapping rules with M3 to control cardinality, and finishes service migration within about a year.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:06Speaker background
- 1:27StatsD challenges
- 4:19Why Prometheus
- 9:20Client-side migration
- 13:57Server discovery setup
- 15:41Cost control
- 18:30User enablement
- 21:38Migration timeline
- 23:22Post-migration metrics
- 24:06Key learnings
- 27:22Q&A
related talks
32mDemocratizing Application Monitoring with Prometheus
25mSponsored Session: Unifying Observability at DoorDash using Open Stand...
29m“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus
23mAutomatically Making Dashboards Load 100X Faster
08mPrometheus in 5 minutes
37mLowering the barrier for ML monitoring
24mPractical Kubernetes Monitoring with Prometheus
37mCloud-native Metric Monitoring with Prometheus