Building Real-Time Data Lake with ClickHouse
summary
Claims compute–storage separation for ClickHouse while keeping real‑time speed by using Project Antalya to query Iceberg on S3 with Parquet files, adding caching layers and a stateless swarm clusters execution model that pre-aggregates remote data. Demonstrates a 1.3 billion‑row benchmark where swarm query latency matches or beats native ClickHouse by provisioning extra compute. Shows an “infinity table” merging hot MergeTree and cold Iceberg data, plans double‑write for real‑time ingest, and provides open‑source Kubernetes manifests plus Terraform for EKS.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:47ClickHouse basics
- 3:34Storage costs and object storage
- 5:21Why ClickHouse is fast
- 8:17Entalia project
- 9:00Swarm execution model
- 11:08Caching and benchmarks
- 12:36Catalog service for Kubernetes
- 13:18Infinity tables
- 14:24Double writes for ingestion
- 15:29Real-time data lake architecture
- 15:51Open source deployment
related talks
34mCatalogs, Iceberg, and Delta: Building the Foundation for Lakehouse Analytics
28mColumnstore Revolution
32mHow open source is re-shaping the cloud data warehouse landscape
40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data Lakes
55mBuilding Fast ClickHouse Analytics for MySQL Transaction Data Using Kubernetes
34mClickHouse® - What is Behind the Fastest Columnar Database?
42mClickHouse: what is behind the fastest columnar database
26mHandling Large Volumes of Immutable Structured Data With ClickHouse