Reinventing ETL for Detection and Response Teams
summary
Claims that traditional security ETL leaves analysts writing 40-line SIEM queries and suffering data decay, illustrated by Luminati’s 8M+ IPs with ~11% daily churn. Proposes doing enrichment in the pipeline using AWS serverless streams, DynamoDB and EDR feeds so events arrive in SIEM already contextualized. Describes three concrete patterns — time travel pattern, telephone pattern — implemented in the open source project substation (3+ years in production, 2+ years OSS) at a fraction of vendor cost.
AI-written summary · about
chapters (11)
- 0:00Introduction to ETL
- 3:04Current State of ETL
- 4:57Problems with Security ETL
- 8:59Data Decay Challenges
- 11:26Improving the Data Pipeline
- 13:09Key ETL Use Cases
- 15:29Time Travel Pattern
- 20:00Telephone Pattern
- 22:46NXR Pattern
- 26:28Conclusion and Q&A
- 27:20Final Questions
related talks
28mPatterns for scalable security detections on AWS
01h 02mSecuring data and democratizing the alert landscape
49mBuilding digital resilience with unified security and observability
50mTo Normalized Logs, and Beyond - Building a Threat Detection...
01h 05mPractical network defense at scale
47mDetection as Code: The Engineering-Focused Future...
12mSecurity data gravity: Visibility & threat detection for data lakes
24mThreat Hunting Powered by Efficient and Straightforward Anomaly Detection on Your Data Lake