A History of Automatic Aggregations
summary
Criteo runs an automatic aggregation layer that drops instance labels and applies counters rates, gauge percentiles and histogram merges to cut cardinality and cost while keeping dashboards and alerts fast. The stack uses Prometheus-compatible agents, remote write to VictoriaMetrics, and optional streaming-agent aggregation; ingestion is ~15 million datapoints/s across ~1 billion active series, and an example app is reduced from 120 million to <4 million series, improving query latency and platform stability.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:24Team objectives
- 3:10Graphite aggregation struggles
- 4:34Overse model
- 7:02Pre-aggregation benefits
- 9:10Overse limitations
- 9:51VictoriaMetrics migration
- 11:14Churn impact on metrics
- 13:09Streaming aggregations
- 14:34Prometheus-native aggregations
- 17:26Key benefits
- 18:08Q&A
related talks
20mAggregating Metrics In-Flight: Challenges, Opportunities
38mAggregating Metrics In-Flight: Challenges and Opportunities
11mAggregating, Alerting and Graphing on Millions of Prometheus T...
23mAutomatically Making Dashboards Load 100X Faster
23mDesigning for High-cardinality Metrics
45mReal Time Analytics with MongoDB
16mTransforming Analytics With AI: A Comprehensive Guide to an Advanced Tech Stack
48mMetrics All the Way: Data Driven Operations