Sweet Streams are Made of These: Applying Metrics-driven Development to Apache Flink
summary
Practical recipe for making Flink production-ready: build a tight metrics cycle using Flink’s REST API and reporters (Prometheus, DataDog) and pair with an external dashboard (Verica example), track Kafka lag per stream, monitor checkpoint size and throughput, and watch for a recurring checkpoint size daily drop that signaled lost state in one case. Use small modular alerts, relabeled alerts routed to Slack, keep bird’s-eye and granular views side-by-side, and expect faster incident response and clearer cross-team ownership.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:25Basic Principles
- 2:15Overview
- 2:57Provider Agnostic
- 3:32DataDriven Development
- 4:07Metrics
- 4:29Flink Dashboard
- 6:19Flink Reporters
- 6:43What else
- 8:11Good dashboard hygiene
- 8:27Example dashboard
- 13:11Accessibility
- 13:52Measuring Everything
- 14:06Antipattern
- 14:21Metrics as a Shared Language
- 14:38Cultural Obstacles
- 14:57Metrics as Shared Language
- 19:06Automation
- 19:42Conclusion
related talks
15mNeedle in the Haystack - Monitoring health of a huge Kafka fleet with Flink
22mEnd-2-End Monitoring and Troubleshooting a Real-Time Data Pipeline
35mLessons learned on Apache Flink application availability in a hosted Flink service
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale
34mAdvanced visualization of Flink and Spark jobs
35mApache Kafka as a Monitoring Data Pipeline
45mTracing: A journey to tactical insights
33mObservability-First Kafka: Engineering Visibility at Scale