Evolution of the Metrics Pipeline: How the Architecture Scaled with Load
summary
Describes how Tinkoff Bank’s observability platform Sage for telemetry evolved as load increased: which components and storage technologies were used to collect, store and analyze metrics, and how the system architecture changed to handle higher ingestion and query volumes.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:09False alerts and load growth
- 3:23Import substitution era
- 5:06First metrics and centralization
- 6:14Monitoring system 2020
- 9:48Storage load problems
- 11:37ClickHouse transition
- 13:05Metrics collector scaling
- 16:01Kafka incident and loose coupling
- 17:29Alerting Engine and global search
- 18:54System stabilization
- 20:20Kubernetes adoption
- 21:44Direct writers and long-term storage
- 23:51VictoriaMetrics cluster
- 26:43Noisy neighbor issues
- 29:30Write latency problems
- 32:39Collector optimization
- 33:43Short vs long-term storage
- 35:13ClickHouse disk nuances
- 36:15Load growth and reliability
- 38:26Key lessons
- 39:08Future plans
- 40:13Observability community
- 41:17Q&A
related talks
42mEvolution of Observability Tools at Pinterest
54m7 Petabytes of Logs in Elastic. How We Did It
42mMetrics at scale
45mMetrics at scale
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
30mThe Challenges of Managing Open Source Infrastructure at Bloomberg
09mKeynote: Scaling Observability Pipelines in the Enterprise
34mThe State of SQL-Based Observability