Beyond Logs: Real Time Anomaly Detection with Machine Learning
summary
Pipeline that ingests Kafka log streams, combines them with batch context in Hazelcast, and runs Python-backed models (SQL or map/journal stages) to score and alert in real time. Uses linear regression and model-aggregating stages, deployable as sidecar or co-located pods. Demonstrated production-scale throughput: 1 billion transactions at <30 ms latency on 45 nodes, linear scaling as nodes add, and outputs to Kafka or files. Code examples show map-based timestamps and IP keys, score thresholds, and live predictions that trigger alerts. Kafka + Hazelcast + Python, map-based trend scoring.
AI-written summary · about
chapters (13)
- 0:00Introduction to Real-Time ML
- 1:03Real-Time ML vs. Traditional
- 3:27Applications and Challenges
- 4:43Combining Data Sources
- 6:57Stream Processing Architecture
- 8:35Introduction to Hazelcast
- 10:43Contextual Data and APIs
- 12:14Scalability and Deployment
- 14:18Python Integration and Scaling
- 16:40Deployment Strategies
- 18:27Log Analysis Demo Setup
- 21:11Running the Pipeline Demo
- 24:38Conclusion and Resources
related talks
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
47mReal-time Fraud Detection Challenges and Solutions
51mReal-time Fraud Detection: Challenges and Solutions
31mRunning Machine Learning Analytics On Traces
40mHow to leverage the Apache Kafka Ecosystem to productionize Machine Learning
19mBig Data
16mAnalysing Real time Activity Streams for Security
01h 21mBuilding a ML Model Observability pipeline