Adaptive and augmented active anomaly detection on dynamic network traffic streams

Abstract

Active anomaly detection queries labels of sampled instances and uses them to incrementally update the detection model, and has been widely adopted in detecting network attacks. However, existing methods cannot achieve desirable performance on dynamic network traffic streams because (1) their query strategies cannot sample informative instances to make the detection model adapt to the evolving stream and (2) their model updating relies on limited query instances only and fails to leverage the enormous unlabeled instances on streams. To address these issues, we propose an active tree based model, adaptive and augmented active prior-knowledge forest (A³PF), for anomaly detection on network traffic streams. A prior-knowledge forest is constructed using prior knowledge of network attacks to find feature subspaces that better distinguish network anomalies from normal traffic. On one hand, to make the model adapt to the evolving stream, a novel adaptive query strategy is designed to sample informative instances from two aspects: the changes in dynamic data distribution and the uncertainty of anomalies. On the other hand, based on the similarity of instances in the neighborhood, we devise an augmented update method to generate pseudo labels for the unlabeled neighbors of query instances, which enables usage of the enormous unlabeled instances during model updating. Extensive experiments on two benchmarks, CIC-IDS2017 and UNSW-NB15, demonstrate that A³PF achieves significant improvements over previous active methods in terms of the area under the receiver operating characteristic curve (AUC-ROC) (20.9% and 21.5%) and the area under the precision-recall curve (AUC-PR) (44.6% and 64.1%).

摘要

主动异常检测通过查询被采样实例的标签,增量更新检测模型,已被广泛用于检测网络攻击。然而,现有方法不能在动态网络流量上实现预期表现,这是因为:(1)它们的查询策略不能采样具有信息量的网络流量,以使检测模型适应数据分布不断变化的网络流量;(2)它们的模型更新仅依赖于有限的查询流量,不能利用网络流量中巨大的未标记流量。为解决这些问题,提出一种自适应增强的主动先验知识森林模型A3PF,用于网络流量的异常检测。通过利用网络攻击的先验知识,寻找能更好区分异常网络流量和正常网络流量的特征子空间,从而构建先验知识森林模型。一方面,为使模型适应不断变化的网络流量,设计了一种新的自适应查询策略,从动态数据分布的变化和异常的不确定性两个方面对具有信息量的网络流量进行采样。另一方面,基于邻域中网络流量的相似性,设计了一种增强更新方法,为查询流量的未标记邻居生成伪标签,从而在异常检测模型更新过程中能够充分利用大量未标记流量。在CIC-IDS2017和UNSW-NB15这两个入侵检测数据集上的大量实验表明,较之相关方法,A3PF性能显著提升。具体而言,其平均AUC-ROC分别提高20.9%和21.5%,平均AUC-PR分别提高44.6%和64.1%。