Tips and Tricks for Developing Streaming and Table Connectors
summary
Practical checklist for building Flink connectors: implement sources and sinks for the Streaming and Table APIs, use Flink’s SerializationSchema/DeserializationSchema and built-in Avro/Parquet/JSON support, package as a shaded connector JAR, and wire metrics for lag and offsets. Support parallelism changes, checkpoints for exactly-once sinks, and credentials/Kerberos for security. Handle event time with per-partition watermarking, consider Flink 1.9’s global watermark tracker or Kinesis tracker, and declare source idleness to avoid stalled watermarks.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:27About Eron
- 1:17Agenda
- 1:54Connector Applications
- 3:16Streaming Connectors
- 9:47Metrics
- 10:34Connector Security
- 11:22Table API
- 12:40Event Time
- 14:33Dynamic Data Sources
- 19:55Flink Watermark Generator
- 21:30Flink Global Watermark Tracker
- 22:38Source Idleness
related talks
51mStream Processing Smackdown: Kafka Streams vs. Apache Flink
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
26m“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ Flink
22mFlink-Kudu connector
40mThe Seven Deadly Sins of Streaming
32mThe easy way to build and run Flink SQL jobs using Cloudera SQL Stream Builder
42mStream Processing with Apache Flink
32mKeynote: Apache Flink - Completing Cloudera’s End to End Streaming Platform