End-to-end Flink SQL with Schema Registry, Kafka & Kudu
summary
An end-to-end Flink SQL pipeline reads Avro records from Kafka via the Schema Registry catalog, enriches by joining on sensor_id to a Kudu table exposed through the Kudu SQL catalog, computes 10‑second tumbling-window averages by sensor type / plant / machine, and writes alerts back to Kafka. Two concrete join implementations are shown: a temporal table join that avoids unbounded state by using processing-time lookups, and a lower-level keyed co-process function that buffers unmatched readings and stores enrichment in keyed state. The pipeline runs and emits per‑window alerts as expected.
AI-written summary · about
chapters (11)
- 0:00Introduction to Cloudera
- 1:33Use case overview
- 3:37Schema registry integration
- 6:15Schema registry demo
- 9:47Apache Kudu overview
- 12:27Kudu catalog demo
- 14:56Challenges in data joining
- 20:43Join solutions
- 23:58SQL join demonstration
- 27:19DataStream API join
- 29:39Conclusion and key takeaways
related talks
36mInteractive Session: Running Flink SQL jobs in production
31mData Pipeline Lifecycle: SQL Everywhere
43mContinuous SQL with Kafka and Flink
29mHigh-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale
43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!
46mFlink SQL in 2020: Time to show off!
49mLet’s Build a Streaming Data Pipeline!
25mOnly SQL: Empower data analysts end-to-end with Flink SQL