StatsHouse: Vkontakte Metrics

on YouTube ↗41m

" summary

StatsHouse describes VKontakte’s metrics pipeline that records about 200 million measurements per second and underpins real-time launch decisions and product/technical choices. It covers the metrics technologies and architecture VK uses to collect, store and operate on those metrics to support live deployments and monitoring.

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 2:40Why monitoring
3. 4:05Prometheus at scale problems
4. 6:13Real-time metrics transfer
5. 8:18Multi-level data aggregation
6. 10:46Historical data processing
7. 13:41System overload protection
8. 17:13Resource budgeting
9. 17:57StatsHouse architecture
10. 20:06ClickHouse data storage
11. 21:32Caching and optics
12. 22:57Interface and visualization
13. 24:02Project open-source announcement
14. 25:24Q&A

" related talks

• 41mHow VK Inserts Data into ClickHouse from Tens of Thousands of ServersYuriy Nasretdinov · HighLoad++ Moscow 2018
• 48mHow VK Inserts Data into ClickHouse from Tens of Thousands of ServersYuriy Nasretdinov · HighLoad++ Siberia 2018
• 54mEverything Everywhere at Once: Tracing under High Load with 10k BackendsAleksandr Kirsanov · Saint HighLoad++ 2023
• 36mVictoriaMetrics: scaling to 100 million metrics per second (EN)Aliaksandr Valialkin · OSMC 2022
• 41mVK Media Recommendations: How We Build the Personalized "For You" FeedS.Malkevych · HighLoad++ 2023
• 24mMetrics-driven developmentLynn Root · EuroPython 2015
• 38mBuilding an Analytics System for a High‑Load Media PlatformO.Novikov, I.Saltanov · HighLoad++ 2014
• 38mGiving Away Your Secrets: Opening Metrics Up to UsersAlexander Ananiadis · SREcon23 Asia/Pacific