One Does Not Simply Query a Stream
summary
Compares concrete ways to query Kafka: Kafka Streams builds key-addressable state stores backed by RocksDB for low-latency key lookups but scales only with topic partitions; Kafka Connect and CDC dump streams into OLTP/warehouse stores for familiar SQL at higher freshness cost; streaming SQL systems (ksqlDB, Materialize, RisingWave, DeltaStream) expose materialized views and push queries for continuous results; real-time analytic DBs like Apache Pinot and ClickHouse power user-facing, multi-dimensional queries at sub-minute freshness while data lakes and warehouses favor batch and higher latency.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:26Query a stream meme
- 3:56Database CDC limits
- 5:42Kafka log query challenges
- 7:10Streaming query types
- 9:18Kafka Connect queries
- 13:54Kafka Streams processing
- 20:22Stream-table duality
- 22:48Streaming SQL databases
- 27:01KSQL DB for Kafka
- 30:58Flink streaming databases
- 32:02Delta Stream SQL
- 33:27Rising Wave overview
- 35:33Pinot real-time analytics
- 39:51Cloud warehouses streaming
- 41:17Data lakes streaming formats
- 43:23Choosing the right tool
- 45:51Community developer experience
- 46:11Summary
related talks
48mOne Does Not Simply Query a Stream
54mHow do you query a stream?
01h 07mKafka Streams and KSQLDB
01h 00mIn Depth Kafka Streams
42mdistributed stream processing for everyone with KSQL #bbuzz
27mNavigating Streaming Infrastructure
35mBuilding a Real-time Streaming Platform Using Kafka Streams...
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!