Lessons learned on Apache Flink application availability in a hosted Flink service

on YouTube ↗35m

" summary

Managed Flink apps run on EKS with RocksDB state backend and S3 for durable storage and availability is measured by refining Flink job state into a job tracking state that separates scheduling, deploying, and running so uptime only counts when all tasks execute user code. Runtime exceptions feed a rule-based exception classifier that labels failures as user, system, or resource and exports labeled downtime via Prometheus → CloudWatch metrics. An auto recovery system that restarts unhealthy task managers reduces cascaded failovers and proves most software faults are recoverable.

AI-written summary · about

" chapters (17)

1. 0:00Intro
2. 1:07Streaming real-time data with AWS
3. 2:48Amazon Kinesis Data Analytics (KDA)
4. 3:30How do you run Apache Flink in Kinesis Data Analytics?
5. 5:55Behind the scenes what the service does . On a Start Application call
6. 7:51Defaults/overrides applied
7. 10:00Apache Flink metrics to measure availability
8. 11:13Why is the measurement noisy ?
9. 13:35Challenges
10. 13:49How to measure uptime/downtime?
11. 18:48How to distinguish user vs. system failures?
12. 24:51How to track classified time?
13. 26:37How to do operation on classified downtime?
14. 28:30Lessons Learned (from engineering) • Failure classification is hard in large and ever-evolving Flink Inspecting and analyzing all the exceptions is challenging
15. 30:28Lessons Learned (from operations) • Downtime not just from infrastructure issues but software
16. 32:36Towards Fast Recovery • Lessons learned are feedback to fast recovery framework
17. 33:24Conclusions • Availability is key metrics for long-running applications

" related talks

• 33mMaintaining highly available Flink applications at scalenagish fanali · Flink Forward Global 2021
• 37mBuilding and operating large-scale streaming applications at KingGyula Fora · Flink Forward Berlin 2017
• 58mBuilding Self Service Platforms for Apache FlinkGorman, Nienhuis, Canzonieri, Novak · Flink Forward Europe 2019
• 36mRunning Flink in production: The good, the bad and the in-betweenLakshmi Rao · Flink Forward Europe 2019
• 16mZero-downtime upgrades of Flink applicationsLakshmi Rao · Flink Forward Global Virtual 2020
• 40mGrowing a managed Flink streaming platform from scratchChen Qin · Flink Forward Global Virtual 2020
• 38mBuilding Unified Streaming Platform at UberZhenqiu Huang, Venkata Sanath Muppalla · Flink Forward Global Virtual 2020
• 21mSweet Streams are Made of These: Applying Metrics-driven Development to Apache FlinkCaito Scherr · Flink Forward Global Virtual 2020