Flink SQL in 2020: Time to show off!
summary
Demo shows a Docker Compose Flink cluster (JobManager/TaskManagers), Hive metastore, Kafka, MySQL and S3 (MinIO) running live and succeeding: a Kafka→S3 CSV snapshot job writes ~50 pages, batch and streaming use the same SQL, and a tumbling window query runs identically in batch and streaming. The demo exercises temporal table joins, enrichment lookups to MySQL, interval joins with five‑minute state, MATCH_RECOGNIZE CEP, materialized views written to Kafka/MySQL with flush 2s/1000 rows, and cites ~25 open FLIPs for ongoing work.
AI-written summary · about
chapters (15)
- 0:00Introduction and overview
- 1:08Streaming SQL basics
- 8:17Demo environment setup
- 14:04Live demo begins
- 17:36Using Hive catalog
- 19:47Snapshot to S3 export
- 21:39Batch vs streaming execution
- 24:15Windowed aggregations
- 25:49Joining tables
- 34:11Pattern recognition
- 37:15Materialized views
- 40:52Current Flink SQL features
- 42:22Future developments
- 45:19Change data capture
- 45:40Summary and closing
related talks
36mInteractive Session: Running Flink SQL jobs in production
43mContinuous SQL with Kafka and Flink
43mLeverage the power and simplicity of SQL on Apache Flink
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobs
25mOnly SQL: Empower data analysts end-to-end with Flink SQL
27mIntroduction to Flink in 30 minutes