Interactive Session: Flink on Autopilot with Splunk
summary
Splunk describes a Fully Managed Autoscaled and Elastic Stream Processing Service that manages ten thousands of Flink jobs by extracting a control plane called Autopilot: a reconciler, message queue and command processor that drives Kubernetes via Maestro and persists pipeline and deployment state. It implements pipeline versioning and safe rollbacks, auto tuning of JVM/memory/network buffers, and an autoscaler that aggregates per-operator metrics, backlogs and task utilization, uses moving averages plus threshold/time-series models and cooldowns, and reports that this design scales successfully (not open source).
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:08Contents
- 3:17Some of Splunk's customers
- 8:04Enter Autopilot
- 11:55Why Autoscaling?
- 13:29Autoscaling Options
- 15:18Why Autoscaling is Hard
- 16:44Autoscaling Algorithm - MAPE
- 17:29Autoscaler Metrics Aggregation
- 18:58Cluster Reports
- 19:59Autoscaler Sources of Information
- 21:16Autoscaler Algorithm
- 21:51Remaining backlog free
- 23:11Task Utilization Algorithm
- 24:15Adjusting the parallelism based off load
- 25:49Lessons Learned
- 27:32Testing
- 28:46Future Work
- 30:41Conclusion
related talks
44mApache Zeppelin: A friendlier way to Flink
39mHow We're Democratizing Stream Processing at Zillow
40mInteractive Session - Live Code Contribution to Apache Flink
36mFlinkspector: Taming the squirrel
27mHow to be data-driven when you aren’t Netflix (or even if you are)
20mProduction Ready Streaming SQL with Ververica Platform
22mI²: Interactive Real-Time Visualization for Streaming Data with Flink & Zeppelin
01h 33mSplunk as a Big Data Platform for Developers