Integrating Flink and Kafka in the standard genomics pipeline
summary
A Flink pipeline replaces the initial genomics steps with two loosely linked modules that stream through Kafka: reader and writer modules exchange many dynamically created topics announced on a control topic and terminated with end‑of‑stream markers. The standard C/C++ aligner is wrapped as a Java library and invoked from Flink. A practical fix uses an event-time hack (artificial timestamps) so count-style windows process final partial windows. Tests on 50 GB compressed input run 11% slower than native on one node and scale nearly linearly; Kafka handled 200M messages in 3 minutes.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:00What is sequencing
- 1:35Applications of sequencing
- 2:49Overview of the process
- 3:35Pipeline overview
- 5:30Models
- 7:17Implementation
- 8:15Alignment
- 11:20Serializer
- 11:55Timestamp
- 12:42Experiments
- 13:11Results
- 13:52Kafka test
- 15:40Conclusion
related talks
38mInteractive Session: Optimising data streaming pipelines on Flink and Kafka
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
36mJoining Infinity: Windowless Stream Processing with Flink
41mBeyond the Hype: When Apache Flink Solves Real Problems
37mConnecting Apache Flink to the World...
43mA Data Streaming Architecture with Apache Flink #bbuzz
30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data Analytics
29mWhen your stream doesn't stream very well