Can a General-Purpose LLM Do Tier-1 Triage?
summary
Controlled experiment evaluates 600 synthetic alerts (three rounds of 100 alerts each, 30% true positives, two hidden campaigns of 15 alerts each) using an agent built with LangGraph, vector search in Chromadb, observability via LangSmith, and schema validation with pydantic. Results show GPT-4 mini gives higher label accuracy but degrades as temporal windows expand, while GPT-5 mini over-aggregates clusters, costs ~46% more and runs ~47% slower yet misses fewer threats; introduces a cost-weighted triage metric (false negative ≈ ×50) to compare tradeoffs.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:46Built AI triage agent
- 4:01Agent architecture
- 6:50Tools and models
- 10:23Realistic alert dataset
- 13:33Tuning and cost insights
- 14:57Experiment with 600 alerts
- 16:21Results and detection
- 18:14Accuracy vs cost tradeoffs
- 21:26False negatives vs positives
- 23:35Cost-weighted scoring
- 25:46Key insights
- 27:58Risks for SOCs
- 30:28Operationalizing AI agents
- 31:54Q&A
related talks
34mDevising and Detecting Phishing: Large Language Models vs. Smaller Human Models
45mHalf hour of labeling power: Can we beat GPT?
16mAutomating Technical Privacy Reviews Using LLMs
38mNext-Gen Detection: Harnessing LLMs for Sigma Rule Automation
37mChallenges and Pitfalls of Applying GenAI in Monitoring
31mBeyond LLMs and Maturing RAG use cases
44mProduction-Grade LLMs & AI Agents
46mHow Large Language Models Are Reshaping the Cybersecurity Landscape