Hello Hydrate! From Stream to ClickHouse with Apache Pulsar and Friends

on YouTube ↗29m

" summary

A working pipeline shows Apache Pulsar, Flink and NiFi streaming device images, MQTT and WebSocket events, CDC and logs into ClickHouse with an offload to object storage. A simple YAML/JSON connector maps a Pulsar topic to a ClickHouse table via JDBC, schema registry enforces JSON/Avro, and a demo imports ~500 records arriving every few seconds without custom code. Connectors are open source with a GitHub template and hub for easy extension. Apache Pulsar ClickHouse Flink

AI-written summary · about

" chapters (7)

1. 0:00Start
2. 1:25The Need For Real-Time Data
3. 2:09What are the Benefits of Pulsar?
4. 5:18A Unified Messaging Platform
5. 6:13Apache Pulsar Overview
6. 7:13Topics
7. 9:37Example: E-Commerce with Pulsar

" related talks

• 39mCodeless Pipelines With Pulsar and FlinkTimothy J Spann · Data Con LA 2021
• 36mHail Hydrate! From Stream to LakeTimothy J Spann · Conf42 Machine Learning 2021
• 47mQuery Pulsar Streams using Apache FlinkSijie Guo · Flink Forward Europe 2019
• 41mDeep Dive into Building Streaming Applications with Apache PulsarTimothy J Spann · CloudOpen @ Open Source Summit Europe 2022
• 25mContinuous Queries on Apache Pulsar using Flink SQLNan Lu · Flink Forward Global 2021
• 35mStreaming Data Made EasyTimothy J Spann, David Kjerrumgaard · OSA Con 2022
• 14mFast Data on-Ramp with Apache Pulsar on K8Timothy J Spann · Kubernetes Batch + HPC Day Europe 2022
• 28mStreaming Lakehouse with Kafka, Flink and IcebergZabeer Farook · apidays Singapore 2025