Metamodelization and Automatic Generation of ML Pipelines
summary
Claims an Apache Flink + Kafka pipeline that serves PMML models delivered via Kafka and auto-generates the Flink job code for enrichment steps. Enrichments are JSON, organized as one Kafka topic per JSON schema, with two flavors: key-based and non-key enrichments; the generator emits Kafka consumers, keyed streams, state, and join functions (automatic generation of keyed joins) so new detection logic deploys faster — compilation and redeploy remain manual and totally new enrichments still need hand coding.
AI-written summary · about
chapters (5)
- 0:00Introduction to instant payments
- 2:59Automating Flink job generation
- 7:19Key enrichment example
- 12:15Other enrichment types
- 13:20Conclusion and limitations
related talks
36mFlink-JPMML: An open source Flink library for Streaming ML Model Serving
21mIntroducing FlinkML
27mData enrichment patterns with Apache Flink
18mAutomate the boring ML stuff with pipelines
38mDistributed Processing for Machine Learning Production Pipelines
17mCrafting Superior ML Pipelines with SapientML
31mExtending Apache Flink stream processing with Apache Samoa ML methods
29mML Engineering Kubeflow