Not So Big – Flink as a true Application Framework
summary
MotaWord uses Flink as an application framework to manage translator workflows and real time invitations rather than as a batch stream processor it runs a single job with 75 operators replicated into 400 tasks keeping state in megabytes and enforcing three constraints no external references during operation most elements shared and everything flows as streams To make broadcasted state queryable a tiny duplicate stream writes a queryable map state and a lightweight REST state server lives in the same repo so the system works in production with acceptable hacky tradeoffs
AI-written summary · about
chapters (14)
- 0:00Introduction and audience
- 1:51Company and Flink use case
- 3:37Flink job complexity
- 8:16Refactoring into business operators
- 12:34Broadcasting state challenges
- 15:52Extracting invitation logic
- 19:47Forwarding state manipulation
- 23:03Abstract business operator
- 25:17State server implementation
- 27:01State type conflicts
- 29:38Duplicate state workaround
- 31:02Summary of approach
- 32:54Future of Flink applications
- 35:25Operational implications
related talks
32mFlink's SQL Engine: Let's open the engine room!
37mKeynote | Flink First: Designing Modern Trading Systems
34mCustom, Complex Windows at Scale using Apache Flink
29mStreaming & Eventing for Top Management
38mBuilding Unified Streaming Platform at Uber
43mStructure and Interpretation of Stream Processing
23mAdvancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets
51mObject-centric processes...