Effortless event filtering: from Java to Flink queries
summary
ING converts Java business logic into Flink SQL queries and shows a live-coded pipeline that reads a Kafka input topic, joins each event against a Kafka compacted lookup table of per-account preferences, filters transactions above the stored threshold and writes matches to an output Kafka topic consumed by an email service. Configuration uses YAML table definitions and the Flink SQL CLI; complex cases use UDFs. Benchmarks show Flink under 800 ms, lower variance and higher throughput than the Java app; production filters ~50% of 500k daily events.
AI-written summary · about
chapters (15)
- 0:00Stream processing concepts
- 2:52Apache Flink engine
- 4:38Flink SQL filtering
- 5:20Flink app internals
- 7:07Flink for real-time
- 9:17Demo setup
- 13:53Kafka sources config
- 16:01Testing Flink SQL
- 17:39SQL script breakdown
- 19:07Java vs Flink SQL
- 20:08Performance Flink vs Java
- 21:52Production data insights
- 22:35Real-world demo
- 23:41Conclusions
- 24:42Q&A
related talks
42mStream Processing with Apache Flink
20mFlink powers Houzz near-realtime pipeline & Kubernetes logging observability
46mKeep it going — How to reliably and efficiently operate Apache Flink
22mBuilding a Kubernetes Operator for Apache Flink in Java
41mWhat If We've Been Scaling Stream Processing Wrong All Along
01h 29mEvent stream processing using Kafka Streams
26mKubernetes & real-time analytics: How to connect these worlds with Apache Flink
30mCloudflow: Spark, Flink, and Akka Working together on Kubernetes