From Streams to Tables and Back Again: A Demo of Flink’s Table & SQL API
summary
Live demo of Flink’s Table & SQL API shows a prototype SQL CLI that runs in embedded and cluster modes, reads a JSON catalog, and registers user-defined table sources and functions. Examples use a New York taxi event stream (each ride emits two events) to run non-windowed aggregates, retractions in dynamic tables, SQL OVER windows and GROUP BY windows (sliding 15‑minute windows every 5 minutes, and a 10‑minute preceding OVER). Running aggregates emit retract messages (false = delete, true = insert), and jobs submit to a cluster and execute, aside from minor live-demo glitches.
AI-written summary · about
chapters (16)
- 0:00Introduction and setup
- 2:31Sequel CLI client overview
- 5:04Taxi events source demo
- 7:51Running simple queries
- 9:23Real-time event processing
- 10:07User-defined functions
- 10:51Running aggregates
- 11:33Filtering and UDFs
- 12:18Over windows example
- 14:25Cluster deployment demo
- 17:17Q&A session
- 17:58Query plan explanation
- 19:14Hive integration discussion
- 20:44Production and sampling goals
- 21:33Future tooling plans
- 22:35State reuse and checkpoints
related talks
43mLeverage the power and simplicity of SQL on Apache Flink
46mFlink SQL in 2020: Time to show off!
39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
22mStream SQL with Flink @ Yelp
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
25mOnly SQL: Empower data analysts end-to-end with Flink SQL
39mChangelog Stream Processing with Apache Flink