How the Metrics Backend Works at Datadog
summary
Datadog’s metrics backend ingests tens of GB/s and billions of timeseries into PiBs of hot and cold storage, using agents, Redis intake queues, Cassandra for tag resolution and multiple Kafka clusters feeding index and point services. Sharding starts by customer id, escalates to metric id for huge customers, and uses a homegrown slicer-based partition balancer to box-fit load. Zonal replicas and per-AZ Kubernetes clusters provide availability; live data migration between clusters remains manual and is being automated.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:08What is a "Data Dog" ?
- 2:11Overview
- 5:46Kafka 101
- 6:35Kafka at Datadog
- 7:16Baby's First Keyfunc
- 8:34Node Failure
- 9:31AZ Failure
- 10:31Kube Cluster Failure
- 11:23Cloud Vendor Failure
- 12:25Remember This
- 13:09Partitioning (not the Kafka kind)
- 13:50Partitioning: Before
- 14:04Partitioning: After
- 16:38Balanced Topic
- 17:19Consumer Shards
- 18:33Big Customers
- 21:21Partition Imbalance
- 24:56Slicer
- 26:45Rebalancing
related talks
35mThe Internals of Metric Store
16mLog Rehydration and Logs to Metrics
24mNew approach to metrics observability for the cloud era
44mChasing the Rainbow: Towards Unified Service Metrics
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
08mNetwork Performance Monitoring
30mThe Challenges of Managing Open Source Infrastructure at Bloomberg
48mMetrics All the Way: Data Driven Operations