The Speed of Metrics, the Fidelity of Traces: Architecting Post-Collection Aggregation
summary
Claims that post-collection aggregation gives metric-like query speed over long windows while keeping raw traces intact, demonstrated with ClickHouse projections that backfill and switch queries to the projection via an explain plan. Example load shows a dashboard query dropping from ~5s to <1s on ~60 million traces after projection, with tradeoffs: extra storage and write-time aggregation cost, need cardinality estimation before creating aggregates, and a move toward automated materialized/accelerated views.
AI-written summary · about
chapters (14)
- 0:00Post-collection aggregation intro
- 1:23Metrics vs raw telemetry trade-offs
- 3:07Aggregation pipeline location
- 5:14Aggregation and high cardinality
- 7:17Post-collection aggregation techniques
- 8:19Traditional vs materialized views
- 11:47ClickHouse tracing example
- 15:59Cardinality estimation before aggregation
- 18:25Aggregation costs and best practices
- 19:29Future of automated aggregation
- 20:13Q&A: Aggregation frequency
- 21:36Q&A: Temporal aggregation
- 23:41Q&A: Time granularity
- 25:25Closing remarks
related talks
20mAggregating Metrics In-Flight: Challenges, Opportunities
38mAggregating Metrics In-Flight: Challenges and Opportunities
33mAggregating metrics in-flight: challenges and opportunities
42mTips & Tricks to Keep Your Queries under 100ms with ClickHouse®
23mAutomatically Making Dashboards Load 100X Faster
19mA History of Automatic Aggregations
17mOptimizing Data-Driven Decisions
26mO11y in One: Exploring ClickHouse as a Unified Telemetry Database