scale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications

on YouTube ↗49m

" summary

Apache Flink is presented as a runtime for stateful stream computations that combines event-time watermarks, large in-process state and exactly-once guarantees via coordinated distributed snapshots. Examples include Alibaba powering realtime shopping and Uber/Netflix streaming analytics; a social-feed built entirely on streams shows full app logic and views computed in Flink. Demo upgrades carry forward live state by snapshot and forks a prior snapshot to reprocess historical data; the redeploy and backfill both run successfully in seconds on an AWS test cluster.

AI-written summary · about

" chapters (17)

1. 0:00Intro
2. 0:24data Artisans
3. 3:37Batch Processing
4. 4:31Apache Flink in a Nutshell
5. 12:10Hardened at scale
6. 15:36Write log and tables
7. 22:26Distributed Memory Image
8. 23:17Stateful Event & Stream Processing
9. 24:31What users built on checkpoints...
10. 27:39Performance
11. 28:35Scaling a Service
12. 29:02Rolling out a new Service
13. 29:58Time: Different Notions of Time
14. 30:54Event Time and Proc. Time Clocks
15. 31:18Time: Watermarks in Parallel
16. 34:34What changes faster? Data or Query?
17. 36:21Blending Analytics & Business Logic

" related talks

• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 42mStream Processing beyond Streaming DataStephan Ewen · ITNEXT Summit 2019
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 25mData: Moving with the flowAlex Walden · Big Data LDN 2022
• 41mBeyond the Hype: When Apache Flink Solves Real ProblemsNaci Simsek · Berlin Buzzwords 2026
• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015
• 43mA Data Streaming Architecture with Apache Flink #bbuzzRobert Metzger · Berlin Buzzwords 2016
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016