How VK Inserts Data into ClickHouse from Tens of Thousands of Servers

on YouTube ↗48m

" summary

Describes VK's production deployment of ClickHouse: why they adopted it, how much data they store, and the methods they use to write/insert data from tens of thousands of servers into ClickHouse. Covers practical operational details of data ingestion at very large scale.

AI-written summary · about

" chapters (11)

1. 0:00ClickHouse intro
2. 3:06Why ClickHouse chosen
3. 7:41Hardware benchmarks
4. 10:12Insertion mechanics
5. 13:23Buffer tables pitfalls
6. 17:39First deployment issues
7. 22:10Kid-in-house proxy improvements
8. 25:27Reverse proxy load balancing
9. 28:38Fasttech protocol memory savings
10. 33:18Lighthouse log interface
11. 35:25Final recommendations Q&A

" related talks

• 41mHow VK Inserts Data into ClickHouse from Tens of Thousands of ServersYuriy Nasretdinov · HighLoad++ Moscow 2018
• 26mHandling Large Volumes of Immutable Structured Data With ClickHouseJim Hague · ACCU 2021 Online
• 31mHow ClickHouse® Inspired Us to Build a High Performance Time Series DatabaseAliaksandr Valialkin · OSA Con 2021
• 51mWhat You Need to Know About ClickHouse ArchitectureAleksey Zatelepin · РИТ++ 2017
• 59mEffective Use of ClickHouseAleksey Milovidov · HighLoad++ 2017
• 55mBuilding Fast ClickHouse Analytics for MySQL Transaction Data Using KubernetesRobert Hodges, Kanthi Subramanian · Percona Live 2022
• 49mSolving real business cases with ClickHouseDmitriy Cherobaev · Saint HighLoad++ 2025
• 42mClickHouse: what is behind the fastest columnar databaseOlena Kutsenko · Berlin Buzzwords 2023