Continuous SQL with Kafka and Flink

on YouTube ↗43m

" summary

Live demo shows continuous SQL against Kafka via Flink: topics created, Avro schemas used, data published from an ADS‑B plane feed and ZIP weather files, joined by fuzzy lat/long and pushed into a materialized view via REST. CDC from Postgres/PostGIS uses Debezium to stream row changes into Flink; inserts route to Kafka, Iceberg or databases. Demo runs in Docker with a GUI tool (NII), scales to thousands of nodes and handles millions of events per second in production.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and overview
2. 4:41Why use Flink for streaming SQL
3. 10:44Real-time data processing with Flink
4. 21:26Demo setup and Kafka integration
5. 27:50Joining streams and materialized views
6. 31:50Change data capture with Flink
7. 36:56Slack integration and AI responses
8. 40:32Summary and Q&A

" related talks

• 46mFlink SQL in 2020: Time to show off!Timo Walther, Fabian Hueske · Flink Forward Virtual 2020
• 27mUnderstanding stream table duality using Kafka, Flink SQL and Debezium formatMuhammet Orazov · Community Over Code EU 2024
• 30mChange Data Capture With Flink SQL And DebeziumMarta Paes · DataEngBytes 2020
• 36mInteractive Session: Running Flink SQL jobs in productionMarton Balassi, Gyula Fora, Matyas Orhidi, Erik Beebe · Flink Forward Global 2021
• 42mChange Data Capture and Processing with Flink SQLJark Wu, Qingsheng Ren · Flink Forward Global Virtual 2020
• 32mThe easy way to build and run Flink SQL jobs using Cloudera SQL Stream BuilderKenny Gorman · Flink Forward Global 2021
• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18