A Cooperative Abnormal Behavior Detection Framework Based on Big Data Analytics

Abstract

As cyber attacks increase in volume and complexity, it becomes more and more difficult for existing analytical tools to detect previously unseen malware. This paper proposes a cooperative framework to leverage the robustness of big data analytics and the power of ensemble learning techniques to detect the abnormal behavior. In addition to this proposal, we implement a large scale network abnormal traffic behavior detection system performed by the framework. The proposed model detects the abnormal behavior from large scale network traffic data using a combination of a balanced decomposition algorithm and an ensemble SVM. First, the collected dataset is divided into k subsets based on the similarity between patterns using a parallel map reduce k-means algorithm. Then, patterns are randomly selected from each cluster and balanced training sub datasets are formed. Next, the subsets are fed into the mappers to build an SVM model. The construction of the ensemble is achieved in the reduce phase. The proposed structure closely delivers a high accuracy as the number of iterations increases. Experimental results show a promising gain in detection rate and false alarm compared with other existing models.