Feature Creation Towards the Detection of Non-control-Flow Hijacking Attacks

Abstract

With malware attacks on the rise, approaches using low-level hardware information to detect these attacks have been gaining popularity recently. This is achieved by using hardware event counts as features to describe the behavior of the software program. Then a classifier, such as support vector machine (SVM) or neural network, can be used to detect the anomalous behavior caused by malware attacks. The collected datasets to describe the program behavior, however, are normally imbalanced, as it is much easier to gather regular program behavior than abnormal ones, which can lead to high false negative rates (FNR). In an effort to provide a remedy to this situation, we propose the usage of Genetic Programming (GP) to create new features to augment the original features in conjunction with the classifier. One key component that will affect the classifier performance is to construct the Hellinger distance as the fitness function. As a result, we perform design space exploration in estimating the Hellinger distance. The performance of different approaches is evaluated using seven real-world attacks that target three vulnerabilities in the OpenSSL library and two vulnerabilities in modern web-servers. Our experimental results show, by using the new features evolved with GP, we are able to reduce the FNR and improve the performance characteristics of the classifier.