Reconciling Kafka, SQL and Parquet
summary
Presents a practical pipeline to export Kafka Avro topics into Parquet files and run instant SQL queries with DuckDB (Avro → Parquet dumps + DuckDB queries), with an optional step to push data into ClickHouse for dashboards while preserving Kafka ACLs and ownership. Describes a proxy and Postgres-protocol shim to enable querying, reports very fast performance (a billion-row Parquet query in seconds and indexed reads dropping from ~4s to ~750 ms versus full topic scans), and notes caveats around metadata handling and concurrent writes.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 3:10Kafka querying problem
- 9:05DuckDB and Parquet
- 16:29PostgreSQL proxy
- 19:18Direct indexing in Kafka
- 22:06Experience and solutions
- 29:09Q&A
related talks
06mA quack at building scalable data pipelines with DuckDB
26mBuilding Data Lake Using DuckDB
01h 07mKafka Streams and KSQLDB
42mBuilding a Data Pipeline with Clojure and Kafka
39mKafka as a Platform: The Ecosystem from the Ground Up
43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
41mKafka as a Platform: The Ecosystem from the Ground Up