A Needle in a Haystack: How to Find a Threat Hidden in Over 6 Billion Logs Per Day
summary
Pipeline ingests over 6 billion cloud-control-plane records per day into durable files, routes them with S3 + SNS + SQS, standardizes varied formats to an agreed schema (example: OCSF standard), stores combined state in OpenSearch for time-based correlation, runs modular detectors that reduce floods to a handful of alerts, applies a separate suppression layer and optional enrichment, runs as independent Kubernetes deployments on spot instances, and proves workable after porting slow Ruby processors to Go for heavy throughput.
AI-written summary · about
chapters (7)
- 0:00Finding threats in cloud logs
- 4:53Unix philosophy for detection
- 13:45Stateful threat analysis
- 20:27Suppressing false positives
- 23:33Scalable runtime components
- 30:52Key takeaways
- 32:14Q&A
related talks
50mTo Normalized Logs, and Beyond - Building a Threat Detection...
40mA needle in a logstack
01h 05mPractical network defense at scale
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
25mThe Importance Of Real time Data In Fighting Cyber Attacks
42mMake your logs work for you
45mUsing time-series data to better understand our products & infrastructure