SoK: Automated TTP Extraction from CTI Reports – Are We There Yet?
summary
The paper systematizes 40 prior methods, reimplements 16 core NLP pipelines, and runs over 100 experiments mapping sentences to MITRE ATT&CK IDs using two public corpora (TrendMicro/ATT&CK subset and Bosch/Anusctr). It shows modern classification and generative models beat rule-based NER in closed-set tests but fail in realistic open-set vs closed-set evaluation where traditional NER stays most robust, revealing a hard performance limit caused by tiny, inconsistent datasets and ontology label ambiguities (e.g., registry edits serving persistence vs stealth).
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:58TTP extraction challenges
- 3:25TTP extraction methods review
- 5:55NLP methods overview
- 8:27Problems with extraction approaches
- 9:11Study design and datasets
- 10:39Closed set performance
- 12:22Open set performance
- 13:06Key bottlenecks
- 14:31Are we there yet?
- 15:14Recommendations and future work
related talks
31mSkynet the CTI Intern: Building Effective Machine Augmented...
41mWhen Knowledge Graph Meets TTPs: Automated & Adaptive Executable TTP Intelligence for Security
13mAPTGen: An Approach towards Generating Practical Dataset Labelled with Targeted...
29mOne Bad OSINT Can Ruin Everything: How Secure is Your CTI RAG System, and Can You Minimize the Risk?
30mProcessing Threat Reports at Scale Using AI and ML: Expectations and Reality
26mDeath to the IOC: What's Next in Threat Intelligence
20mThe Power of Words: Generating PowerShell Attacks from Natural Language
26mImproving Cybersecurity Posture with an AI Security Assistant