Beaming Flink to the Cloud @ Netflix

on YouTube ↗34m

" summary

Netflix runs a Stream Processing as a Service that uses Kafka, Flink and Apache Beam to move and transform live events at massive scale, claiming 700 billion events/day and about 1.3 PB/day of handled data. The system auto-provisions Kafka topics, dashboards and alerts, runs multi-tenant jobs in containers, and achieves four nines availability with migrations that keep the gap under <0.1% data loss.

AI-written summary · about

" chapters (7)

1. 0:00Netflix event infrastructure
2. 3:01Keystone architecture
3. 7:13Routing and deployment
4. 13:41Transitioning to Flink
5. 19:29Titus and Flink integration
6. 23:33Performance and enhancements
7. 29:32Future directions and QA

" related talks

• 22mNetflix Data Mesh: Composable Data ProcessingJustin Cunningham · Flink Forward Virtual 2020
• 25mData: Moving with the flowAlex Walden · Big Data LDN 2022
• 49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing ApplicationsStephan Ewen · Scale By the Bay 2017
• 43mA Data Streaming Architecture with Apache Flink #bbuzzRobert Metzger · Berlin Buzzwords 2016
• 20mBuilding metric platform using Flink for massive scale at NetflixAbhay Amin · Flink Forward Virtual 2020
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 19mBuilding Stateful Streaming PipelinesAnkit Jhalaria · Flink Forward Virtual 2020
• 45mApache Flink — Real-Time Data ProcessingGaël Renoux · Devoxx France 2024