Interactive Session - Live Code Contribution to Apache Flink
summary
Flink repo with over a quarter billion lines gets a concrete feature: add an executeAndCollect API that returns a closable iterator and a bounded collect variant that cancels the job after N elements. Implementation reuses task-manager-to-job-manager accumulators by piggybacking on heartbeats to stream records back with minimal overhead. Development uses IntelliJ, JUnit/Hamcrest, a mini cluster with client resource for networked tests, and Azure Pipelines; tests compile, run against the mini cluster, and pass locally, proving the approach works.
AI-written summary · about
chapters (17)
- 0:00Introduction to Flink contributions
- 3:53Contribution process and guidelines
- 7:47Exploring Flink codebase structure
- 11:42Live code contribution setup
- 15:54Refactoring and moving classes
- 18:07Adding Apache license and annotations
- 20:16Implementing executeAndCollect method
- 23:56Adding limit functionality
- 27:10Input validation and error handling
- 31:24Code style and checkstyle fixes
- 32:30Overloaded method for convenience
- 33:12Writing tests for new features
- 36:15Running and validating tests
- 36:59Deprecating old experimental API
- 37:43Scala API compatibility
- 38:48Documentation and PR submission
- 39:10Q&A and closing remarks
related talks
44mApache Zeppelin: A friendlier way to Flink
38mInteractive Session: Flink on Autopilot with Splunk
36mFlinkspector: Taming the squirrel
40mKeynote: How Can We Take Flink Forward?
50mKeynote
06mInterview with Ted Dunning
22mI²: Interactive Real-Time Visualization for Streaming Data with Flink & Zeppelin
44mThe Art of Refactoring