Running Flink in production: The good, the bad and the in-between
summary
Lyft describes how it runs Apache Flink in production to power features like pickup ETA, dynamic pricing, ML features for fraud detection, and real-time analytics. The talk covers the platform architecture and deployment model—distinguishing native Flink applications from analytics run on Dryft—and discusses operational challenges at scale such as scaling Flink connectors, handling event-time skew and source synchronization, common failure patterns, and plans to re-architect the streaming platform onto Kubernetes.
AI-written summary · about
chapters (22)
- 0:00Intro
- 2:16Data Platform @ Lyft
- 3:23Abstraction levels
- 7:03Creating a new Flink Job
- 9:09Deployment and tooling
- 11:37Production stories
- 12:07Integrations
- 12:42Balancing shard assignments
- 14:20Faster and adaptive reads
- 16:10Event time skew in source partitions
- 18:12Source Synchronization
- 20:08Flink contributions
- 20:29Checkpointing
- 23:18Challenges
- 25:46Learnings
- 27:09Flink on ks: Motivation
- 28:37Flink on ks: Overview
- 29:51Flink Application CRD
- 30:36Flinkk8soperator: Design goal
- 30:55State Machine
- 32:42Flink on k8s: Wins
- 33:45Open Source
related talks
50mPanel: Large Scale Streaming in Production
16mZero-downtime upgrades of Flink applications
38mBuilding Unified Streaming Platform at Uber
35mLessons learned on Apache Flink application availability in a hosted Flink service
44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & Beyond
36mBuilding a Self-Serve Stream Processing Platform with Flink
37mBuilding and operating large-scale streaming applications at King
40mGrowing a managed Flink streaming platform from scratch