How We Scaled Metrics Monitoring for 10,000 Applications With Victoria...
Talkat Open Source Summit Japan + AI_dev: Open Source GenAI & ML Summit + Automotive Linux Summit 2025 · 2025-12-08..12-10·
summary
Migrates a Prometheus+Thanos stack to a VictoriaMetrics cluster and VM Agent fleet to handle 1.6M scrape targets for 10,000 apps and ingest ~50M samples/sec, replacing object storage with block storage and saving ~50% total cost. Builds centralized scrape-management, uses Kafka + Flink streaming aggregation to pre-aggregate 300M samples every 2 minutes, and a PromQL rewrite proxy that routes rewritten queries to aggregated series. Results: ~10x faster queries (99th ~300ms), 90% fewer missing-sample/OOM incidents.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:05Migrating from Thanos to VictoriaMetrics
- 6:24Scaling metrics with VMagent
- 10:39Centralized scrape targets
- 14:10Pre-aggregation for queries
- 18:50Hiding aggregation complexity
- 20:15Key outcomes and architecture
- 20:59Q&A
- 22:25Comparison with Cortex and Thanos
- 24:14Prometheus vs VictoriaMetrics semantics
- 26:44Target discovery trade-offs
- 30:04Final Q&A
related talks
40mScale Your Monitoring Solution With the VictoriaMetrics Ecosystem
36mVictoriaMetrics: scaling to 100 million metrics per second (EN)
01h 00mHow do We Keep Metrics for a Long Time in VictoriaMetrics
11h 43mPercona Community Live Conference - Day 1 - Open Source Databases Conference
33mAggregating metrics in-flight: challenges and opportunities
31mHandling Billions of Metrics with Prometheus and Thanos
34mSpecifics of Data Analysis in Time Series Databases
50mHow To Improve PostgreSQL Monitoring and Observability With PMM