Threat Hunting Powered by Efficient and Straightforward Anomaly Detection on Your Data Lake
summary
Serverless SQL pipeline reads each source table once, aggregates JSON-defined counters into dedicated tables and flags anomalies with simple rules (value > mean + 2σ), linear trend via reg_slope, and pattern search with match_recognize using Trino. The system runs on a petabyte-scale data lake (terabytes added daily, billions of records hourly), produces ~20 million anomalies from hundreds of counters, caught a 4.7M req/s DDoS, and supports LLM-generated SQL queries for natural-language anomaly search.
AI-written summary · about
chapters (9)
- 0:00Introduction to threat hunting
- 1:08Data Lake core technology
- 5:31Challenges in anomaly detection
- 8:24Real-world anomaly examples
- 12:20Solution details overview
- 14:06Data flow and SQL-based detection
- 16:13Anomaly detection with SQL functions
- 21:14Browsing anomalies with LLM
- 22:39Summary and benefits
related talks
50mHunting Supply Chain Threats Using Anomaly Detection
42mThreat Detection across all environments with SnowflakeData Security Lake
48mHunting Stealth Adversaries with Graphs & AI
50mTo Normalized Logs, and Beyond - Building a Threat Detection...
29mThresholds Are for Old Threats: Demystifying AI and Machine Learning to Enhance SOC Detection
30mBetter data quality with ML & visualization
29mReinventing ETL for Detection and Response Teams
52mDevil in the Haystack