Five-sigma Network Events (and How to Find Them)
summary
Code implements anomaly detectors in Python using HyperLogLog sketches, incremental mean/std and online linear regression to track per-IP cardinalities and slopes. It processes ~5 million netflows per minute on one CPU and after 10 million flows flags one persistent port scanner, one current explosion, seven previously pegged hosts and roughly half of frozen hosts later unfrozen. Assumes Gaussian thresholds (five-sigma); empirical distributions show long tails, KS tests versus Poisson and power-law make Gaussian the least-bad fit but still imperfect.
AI-written summary · about
chapters (12)
- 0:00Start
- 1:05Finding the Strange & Unusual
- 2:18Anomaly & Outlier Tools
- 3:18Using Python
- 4:11Creating Tools for Outlier Detection
- 5:09Discover Bad Things Before Big Problems
- 5:47Standard Deviation
- 6:52Project Overview
- 10:06Tools To Use: Sketching & Streaming
- 14:53Explosion Detector
- 15:28Host Stabilization
- 23:34Conclusions
related talks
01h 10mPrincipled Performance Analytics
49mRelational Observability for Cloud-Native Security and Data Science
29mPractical ISP CSIRT Incident Handling w/ Network Flows, ELK Stack & Cybersecurity Intelligence...
25mGraph-based ML Anomaly Detection and Insights for Envoy Systems
30mDebugging at Scale Using Elastic and Machine Learning
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
27mLog messages processing using NLP tools
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning