AttacKG: Constructing Technique Knowledge Graph from Cyber Threat Intelligence Reports

Abstract

Cyber attacks are becoming more sophisticated and diverse, making attack detection increasingly challenging. To combat these attacks, security practitioners actively summarize and exchange their knowledge about attacks across organizations in the form of cyber threat intelligence (CTI) reports. However, as CTI reports written in natural language texts are not structured for automatic analysis, the report usage requires tedious manual efforts of threat intelligence recovery. Additionally, individual reports typically cover only a limited aspect of attack patterns (e.g., techniques) and thus are insufficient to provide a comprehensive view of attacks with multiple variants.

In this paper, we propose AttacKG to automatically extract structured attack behavior graphs from CTI reports and identify the associated attack techniques. We then aggregate threat intelligence across reports to collect different aspects of techniques and enhance attack behavior graphs into technique knowledge graphs (TKGs).

In our evaluation against real-world CTI reports from diverse intelligence sources, AttacKG effectively identifies 28,262 attack techniques with 8,393 unique Indicators of Compromises. To further verify the accuracy of AttacKG in extracting threat intelligence, we run AttacKG on 16 manually labeled CTI reports. Experimental results show that AttacKG accurately identifies attack-relevant entities, dependencies, and techniques with F1-scores of 0.887, 0.896, and 0.789, which outperforms the state-of-the-art approaches. Moreover, our TKGs directly benefit downstream security practices built atop attack techniques, e.g., advanced persistent threat detection and cyber attack reconstruction.

Appendices

A Selecting the Threshold Value

The selection of the threshold value for node/graph alignment scores affects the accuracy and efficiency of AttacKG. Specifically, too low a threshold for graph alignment score could result in premature matching (false positives), while too high could lead to missing reasonable matches (false negatives). For node alignment score, too low a threshold could leave unnecessary alignment candidates and cost longer report analysis time, while too high could lead to false negatives. Thus, there are trade-offs in choosing optimal threshold values. To determine optimal threshold values, we measure the F-score and report analysis time using varying threshold values, as shown in Fig. 5, and select optimal threshold values (0.65 for node alignment, 0.85 for graph alignment) that make each index better at the same time.

Fig. 5.

Threshold selection

B Ablation Study of AttacKG

In particular, we first remove part of the attributes in entities: the IoC information and natural language text termed $AttacKG _{w \setminus o\ IoC\ information}$ and $AttacKG _{w \setminus o\ natural\ language\ text}$, respectively. Note that unlike the EXTRACTOR’s practice of merging entities, which may result in information loss, we only remove partial entity attributes without sacrificing the structural information of attack graphs. Moreover, we obtain another variant by filtering dependencies in attack graphs termed $AttacKG _{w \setminus o\ dependencies}$. That is, we predict attack techniques only based on entity sets. Finally, we disable the graph simplification component termed $AttacKG _{w \setminus o\ graph\ simplification}$.

As different component combinations may affect the distribution of alignment scores, we adjust and choose identification thresholds separately for AttacKG variants in light of the optimal F1-scores. The results are summarized in Table 6. We find that removing any component would degrade AttacKG ’s performance, which well justifies our design choice. Especially, $AttacKG _{w \setminus o\ dependencies}$ consistently performs the worst across all evaluation metrics. It verifies the substantial influence of graph structures in technique identification.

Table 6. Ablation study of different components used in technique identification.

C Efficiency of AttacKG

Settings. We experimentally compared AttacKG’s efficiency with TTPDrill and Extractor on the 16 CTI report samples mentioned in Sect. 4.1 on a PC with AMD Ryzen 7-4800H Processor 2.9 GHz, 8 Cores, and 16 Gigabytes of memory, running Windows 11 64-bit Professional. The size of the reports used as samples ranges from 61 words to 1029 words, with an average of 278.2 words.

Results. Extractor is the most complex system that consists of multiple NLP models and thus has the highest runtime overhead, taking 239.70 s on average to parse a report. Compared to Extractor, AttacKG adopts a simpler CTI report parsing pipeline. On average, it takes 8.9 s and 15.1 s for graph extraction and technique identification, respectively, totaling 24.0 s. TTPDrill, on the other hand, uses the simplest model without constructing attack graphs and thus is the fastest, taking only 5.9 s on average per report, but at the cost of a high false-positive rate.