Towards a Flink Autopilot in Ververica Platform
summary
Ververica Platform Autopilot automatically reconfigures Flink jobs by combining operator idle time ms/sec with source Kafka lag to compute exact parallelism changes instead of blind factors, minimizing expensive savepoint rescaling and preferring to scale whole pipeline so slots stay balanced. An experiment shows rescaling from 2 to 3 parallelism when lag growth implies three records/second backlog, cutting ingestion delay from hours to near real time. Roadmap adds vertical scaling, faster online checkpoints, Flink 1.12 source support and managed configuration tuning.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:37Do we need an Autopilot for streaming applications?
- 3:07Scaling Stages
- 5:07Granularity of Rescaling
- 7:05Utilization Indicators
- 8:48idleTimeMsPerSecond Limitations
- 9:40Possible Solutions
- 13:28Case Study
- 15:31Ververica Platform Autopilot Cockpit
- 16:11Ververica Platform Autopilot + SQL
- 16:28Future Work
related talks
16mFrom Zero to Production with Ververica Platform
43mDynamic Scaling - How Apache Flink adapts to changing workloads
37mThe Future of Apache Flink
36mSources, Sinks, and Operators: A Performance Deep Dive
25mFlink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8s
40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobs
41mRunning Apache Flink everywhere
47mBring Flink to Large Scale Production