Aggregating, Alerting and Graphing on Millions of Prometheus T...
summary
Shows that M3 aggregation already turns very high-cardinality counters into smaller derived metrics at ingest so alerts and dashboards stay fast, claiming it handles 30 million time series workloads by distributing multi-step aggregations in memory across nodes and writing aggregates back to storage. Works with Prometheus remote_write receivers and M3 v1.3, acts as a supplement to recording rules rather than a replacement, avoids expensive reverse-index queries, and reduces load on TSDBs so alert evaluation completes on time.
AI-written summary · about
chapters (8)
- 0:00Start
- 1:23Millions of timeseries? Why?
- 3:24What if we remove the pod cardinality?
- 3:51Recording rules to the rescue
- 4:26Recording rules very high cardinality issue 1 Rule group evaluations errors
- 5:57Attempting recording rule workarounds
- 8:42M3 aggregation
- 10:33And this was just a single metric
related talks
37mM3 and Prometheus, Monitoring at Planet Scale for Everyone
31mLong-Term Metrics with M3 and Prometheus
23mAutomatically Making Dashboards Load 100X Faster
20mAggregating Metrics In-Flight: Challenges, Opportunities
23mAuto-suggesting and generating recording rules for Prometheus
19mStream vs. Batch: Leveraging M3 and Thanos for Real-Time Aggregation
38mAggregating Metrics In-Flight: Challenges and Opportunities
33mAggregating metrics in-flight: challenges and opportunities