Change data capture in production with Apache Flink
summary
OVH builds a CDC pipeline that reads MySQL/Postgres binlogs with Go collectors into Kafka, processes and normalizes events with Apache Flink on YARN (watermarks, custom windows, checkpoints, side outputs) and writes ORC files into a Kerberized HDFS/Hive stack; schema translation uses a pivot format, state runs in RocksDB for state and the system handles 10 million events per day and about 2.5k tables synced (some tables >1B rows). The pipeline is production and effective but requires manual schema-change work and aims to move to Hive 3.
AI-written summary · about
chapters (11)
- 0:00Introduction to OVH and CDC
- 3:15Database replication challenges
- 9:13BI team tools and Hive integration
- 12:24Handling multiple database types
- 17:19Flink pipeline design and Kafka setup
- 20:54Error handling and checkpoints
- 23:41Metrics and monitoring
- 27:33Hive metadata and transactional tables
- 31:55Production deployment and scaling
- 35:47Future improvements and open source plans
- 36:08Q&A session
related talks
42mChange Data Capture and Processing with Flink SQL
15mPowering OLAP at Uber using Apache Flink
29mData Acquisition: Overcoming Challenges
31mFrom Warehouse to Lakehouse: Data at Rest
37mData Contracts In Practice With Debezium and Apache Flink
42mStream Processing with Apache Flink
32mKeynote: Apache Flink - Completing Cloudera’s End to End Streaming Platform
31mEnd-to-end Flink SQL with Schema Registry, Kafka & Kudu