Zero-downtime upgrades of Flink applications
summary
Lyft moves stateful Flink jobs from EC2 to Kubernetes and implements Flink Kubernetes Operator-driven blue‑green upgrades that create a new cluster, take a savepoint of the running job without canceling it, and submit the savepoint to the new cluster so both jobs run in parallel. A lightweight DynamoDB config store records the production primary; an sp/jsonnet CLI drives updates. Costs include duplicate processing and extra Kinesis shards, but the workflow works in production and is contributed to the open source operator.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:36A brief history
- 1:36Deployment
- 5:18Requirement (1)
- 5:42(1) Blue-Green deploys
- 8:08Requirement (2)
- 9:51Other Considerations
- 14:44What's next
related talks
26mKubernetes & real-time analytics: How to connect these worlds with Apache Flink
42mManaging Apache Flink on Kubernetes - FlinkK8sOperator
37mSpark and Flink Running Scalable in Kubernetes #bbuzz
36mRunning Flink in production: The good, the bad and the in-between
25mFlink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8s
33mMaintaining highly available Flink applications at scale
22mBuilding a Kubernetes Operator for Apache Flink in Java
35mLessons learned on Apache Flink application availability in a hosted Flink service