DataEngBytes 2023 - MEL-T1-08 - Troy Sellers

on YouTube ↗27m

" summary

Shows an implemented anomaly detector using Apache Flink and Kafka running on Aiven managed services that flags IoT devices whose CPU usage averages above 80% CPU threshold over a 30-second window and publishes alerts to a Kafka topic. Uses Flink SQL and stateful operators, handles out-of-order events with event-time watermarks (example waits 10 seconds for late events), relies on checkpoints and savepoints for fault-tolerant recovery, and provides Terraform scripts to deploy the pipeline.

AI-written summary · about

" chapters (4)

1. 0:00Welcome
2. 3:12Anomaly detection with Flink
3. 15:25Flink concepts
4. 24:55Live demo and resources

" related talks

• 46mData Analytics-Based Detection At ScalePedro Chaves, Jan Pospisil · BSides Lisbon 2018
• 43mA Data Streaming Architecture with Apache Flink #bbuzzRobert Metzger · Berlin Buzzwords 2016
• 45mApache Flink — Real-Time Data ProcessingGaël Renoux · Devoxx France 2024
• 21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & CassandraCiesielczyk, Zontek · Flink Forward Europe 2019
• 31mEnd-to-end Flink SQL with Schema Registry, Kafka & KuduGyula Fora, Balazs Varga · Flink Forward Global Virtual 2020
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 39mMastering real-time anomaly detection with open source toolsOlena Kutsenko · Berlin Buzzwords 2025
• 29mIntelligent Log Analysis and Real-time Anomaly Detection @ SalesforceAndrew Torson · Flink Forward Europe 2019