Mastering real-time anomaly detection with open source tools
summary
Builds a streaming pipeline that trains and retrains time‑series models on historical data in Apache Iceberg, ingests live events via Apache Kafka, processes sliding windows and runs detection in Apache Flink, and writes a boolean anomaly flag back to Kafka. Implements a lightweight ARMA forecast (pmdarima, three‑sigma thresholds) for fast baseline alerts and an LSTM reconstructor for complex patterns (much higher training cost). Demo on synthetic data injects red anomalies into blue normals and the pipeline reliably flags them, illustrating ARMA vs LSTM tradeoffs and the role of Apache Iceberg for model refinement.
AI-written summary · about
chapters (12)
- 0:00Introduction to anomaly detection
- 1:26Types of anomaly detection
- 8:12Predictive analytics for anomalies
- 10:40Statistical models for anomalies
- 15:23ARMA model explained
- 21:59Real-time streaming data setup
- 25:14Implementing ARMA with Kafka and Flink
- 28:07LSTM model for anomaly detection
- 32:33Implementing LSTM with Kafka and Flink
- 35:03Choosing the right model
- 37:13Combining statistical and ML models
- 38:37Q&A session
related talks
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
29mREAL-TIME ANOMALY DETECTION MADE EASY
43mDetecting Anomalies in Streaming Data
29mAI-Powered Anomaly Detection in Cloud Data Pipelines
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
44mAnomaly Detection in Telecom with Spark
16mAnalysing Real time Activity Streams for Security
19mThe Rise of AI in Anomaly Detection