Beyond Logs: Real-Time Anomaly Detection with Machine Learning
summary
Claim: logs and traces are stored together in a scalable repository and scored with ML to surface anomalies within ~50 ms. Implementation shows two patterns: a DIY stack using Kafka or Apache Pulsar for ingestion, Flink + Redis + online feature store for enrichment and streaming, and a simpler single‑jar fast data store + stream engine (open source on GitHub) that runs on a laptop. Demo uses sliding windows, SQL maps, per‑event scoring and a join with historical maps to explain why alerts fire.
AI-written summary · about
chapters (19)
- 0:00Real-time anomaly detection intro
- 1:50Context in log analysis
- 3:59Sub-millisecond apps
- 5:24Batch vs real-time processing
- 7:08ML challenges in real-time
- 8:12Scaling under load
- 10:02Process logs before DB
- 11:47Microservices for real-time
- 14:19Real-time log analysis stages
- 16:04DIY stream processing pitfalls
- 19:17One jar for real-time
- 22:27Vector databases
- 24:14Demo setup
- 27:07Log generation & processing
- 30:16Context for failure detection
- 34:13Scaling to billions
- 35:56Real-time ML infrastructure
- 37:43Resources & next steps
- 38:06Q&A
related talks
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
21mAnomaly Detection for Platform Logs using Embedded Context
31mRunning Machine Learning Analytics On Traces
16mAnalysing Real time Activity Streams for Security
30mDebugging at Scale Using Elastic and Machine Learning
39mMastering real-time anomaly detection with open source tools
44mAnomaly Detection in Telecom with Spark
31mAnomaly Detection on OpenStack Logs using M.L.