Advanced visualization of Flink and Spark jobs
summary
A real time visualizer for Apache Flink and Spark exposes the physical plan by instrumenting metrics, extending REST APIs, and sampling key distributions with a counter-based, decay-and-backup strategy. It surfaces an input-channel level matrix, shows top-5 heavy keys, and reports local versus remote shuffle fetches. A dynamic partitioning optimizer uses those signals to redirect heavy keys and cuts typical job runtime about 2x and up to 5–6x when hot keys collide. Spark supports historical replay via the history server, Flink currently lacks a history server.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:40Data skew
- 2:11Hungarian Academy of Sciences
- 3:19Use cases
- 4:29Phelps effect
- 6:25Use case
- 7:57Flink vs Spark
- 9:49Shuffle Phase
- 12:49Summary
- 21:42Visualization
- 25:06Dynamic partitioning
- 29:28Questions
related talks
30mOptimize Spark Jobs Using Its UI
39mSparklens: understanding the scalability limits of Spark applications
29mApache Spark Performance Tuning Session with Delight
44mProcessing large data with Apache Spark
35mLessons learned on Apache Flink application availability in a hosted Flink service
21mSweet Streams are Made of These: Applying Metrics-driven Development to Apache Flink
15mNeedle in the Haystack - Monitoring health of a huge Kafka fleet with Flink
22mI²: Interactive Real-Time Visualization for Streaming Data with Flink & Zeppelin