Streamlining Real-time Data Analytics: Integrating ClickHouse with Apache Kafka
summary
Shows a concrete ClickHouse+Apache Kafka pipeline: Kafka Engine table + materialized view inserting into MergeTree, named collections for brokers, and config knobs like kafka_max_block_size and kafka_skip_broken_messages. Demonstrates a laptop Docker demo that sustains bursts up to 200k rows/sec then settles to production pace, discusses partitioning, consumer-group rebalances, schema-evolution strategies, write‑to‑Kafka limits, and practical failure modes with diagnostics and mitigation steps.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:51ClickHouse and Kafka essentials
- 5:23Why ClickHouse for analytics
- 8:18Apache Kafka decoupling
- 11:29Kafka and ClickHouse with Docker
- 13:56ClickHouse tables for Kafka
- 16:47Kafka tables in ClickHouse
- 18:55Streaming with materialized views
- 20:20Querying real-time data
- 23:48Optimizing read speed
- 25:14Kafka partitions and groups
- 27:21Schema evolution handling
- 28:45Scaling for production
- 31:59Rebalancing impact
- 33:21Writing to Kafka from ClickHouse
- 34:27Alternative Kafka integration
- 35:30Exactly-once processing
- 37:17Error handling and monitoring
- 38:42Key takeaways and closing remarks
related talks
55mBuilding Fast ClickHouse Analytics for MySQL Transaction Data Using Kubernetes
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
41mKafka as a Platform: The Ecosystem from the Ground Up
02h 05mA Deep Dive into Apache Kafka This is Event Streaming
49mLet’s Build a Streaming Data Pipeline!
17mBuilding Real-Time Data Lake with ClickHouse
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
39mKafka as a Platform: The Ecosystem from the Ground Up