DrSquirrel - A self-service diagnosis tool for Flink jobs

on YouTube ↗22m

" summary

DrSquirrel aggregates Flink metrics and logs into per-job snapshots (metrics reporter, Kafka log4j appender, a Flink job watcher that emits snapshots to Kafka and a stateless server/UI) and runs health checks that surface back pressure and checkpoint timeout causes, offer actionable fixes, and expose configs queryable in Presto/Hive. It monitors 100% of test and prod jobs, cuts issue detection from hours to <5 minutes, detects ~90% of common issues and halves platform support load while enabling CI/CD gating and future auto-tuning.

AI-written summary · about

" chapters (7)

1. 0:00Intro
2. 1:40Stream Processing Platform (SPP) Overview
3. 4:33Challenges in Flink Adoption
4. 6:31Pain Points in Troubleshooting
5. 10:22Dr. Squirrel Feature Walkthrough
6. 18:06Dr. Squirrel Architecture
7. 20:17Future Work

" related talks

• 36mFlinkspector: Taming the squirrelAlexander Kolb · Flink Forward Berlin 2016
• 44mApache Zeppelin: A friendlier way to FlinkTrevor Grant · Flink Forward Berlin 2016
• 20mProduction Ready Streaming SQL with Ververica PlatformFabian Hueske · Flink Forward Global Virtual 2020
• 27mHow to be data-driven when you aren’t Netflix (or even if you are)Charles Sonigo · Demuxed 2018
• 28mThink Backstage! Beyond PluginsDebabrata Panigrahi, Jenil Jain · BackstageCon Europe 2026
• 45mA Practical Guide to DashboardingJessica White · NDC Porto 2019
• 35mTouK Nussknacker – creating Flink jobs with GUIMaciek Próchniak · Flink Forward Berlin 2017
• 40mInteractive Session - Live Code Contribution to Apache FlinkSeth Wiesman · Flink Forward Global Virtual 2020