Mastering real-time anomaly detection with open source tools
summary
Builds a live pipeline using Apache Kafka for ingestion, Apache Flink for windowed processing, and ARIMA/ARMA plus LSTM for forecasting and anomaly scoring; uses Apache Iceberg for historical storage and periodic retraining. Implements fast statistical layers (three‑sigma / z‑score, 99.7% rule) as a first filter, ARMA with differencing for seasonal/time‑series forecasts, and pretrained LSTM for complex patterns, reporting anomalies back to Kafka and retraining models on rolling windows.
AI-written summary · about
chapters (8)
- 0:00Introduction to anomalies
- 2:01Types of anomalies
- 6:36Evolution of detection
- 10:46Streaming infrastructure
- 15:59Threshold-based methods
- 20:13The ARMA model
- 30:13LSTM deep learning
- 36:45Choosing the right method
related talks
30mMethods & Tools For Time Series Data Science Problems With InfluxDB
30mAnomaly Detection with Apache Pinot & ThirdEye
47mBuilding High Performance Real-time Stream Processing Microservices
45mAnomaly Detection in Time Series
47mReal-time Fraud Detection Challenges and Solutions
32mBuilding Real-time Systems with Open Source Technologies
14mPerformance Optimization in High-Volume Transaction Systems
29mData Acquisition: Overcoming Challenges