Flinking, Fast and Slow

on YouTube ↗20m

" summary

Built the company’s first Flink service in production that reads from Kafka, runs two Flink jobs (map/reduce then watermarking), writes to Kafka and an internal datastore on a daily cadence, and successfully scales to high throughput. Adoption requires slowing rollout: invest a dedicated Flink expert, redefine ownership of upstream/downstream monitoring, provide internal advocate program (blog posts, talks, Slack support), and add code for savepoint storage and container/image hosting to meet operational needs.

AI-written summary · about

" chapters (9)

1. 0:00Introduction and background
2. 1:07Early adoption challenges
3. 3:15Use case and architecture overview
4. 5:41Integration challenges and solutions
5. 8:12Building internal flink advocates
6. 11:22Operational needs and infrastructure
7. 13:49Expertise and resource allocation
8. 16:20Impact of flink’s speed on ecosystem
9. 18:29Conclusion and Q&A

" related talks

• 45mThe State of Flink and how to adopt Stream ProcessingStephan Ewen · Flink Forward Berlin 2017
• 38mFrom Flink 1.3 to 1.4Till Rohrmann · Flink Forward Berlin 2017
• 30mBlink: Improvements to Flink & its application in Alibaba SearchXiaowei Jiang · Flink Forward Berlin 2016
• 35mBeyond the Watermark: On-Demand Backfilling in FlinkMaxim Fateev · Flink Forward Berlin 2016
• 38mBridging the Chasm Between Research & Software DevelopmentLinda Stougaard Nielsen · GOTO Copenhagen 2022
• 47mAll the DataOps, all the paradigmsLars Albertsson · Berlin Buzzwords 2025
• 34mBuilding High Resiliency into Digital SolutionsSuresh Reddy · Developer Summit 2020
• 42mColliding worlds or bending Spark towards enterprise needsBoris Trofimov · JavaDay Kharkiv 2016