Abstract
Intrusion detection systems (IDSs) are devices or software applications that monitor networks or systems for malicious activities and signals alerts/alarms when such activity is discovered. However, an IDS may generate many false alerts which affect its accuracy. In this paper, we develop a cyberattack triage algorithm to detect these alerts (so-called outliers). The proposed algorithm is designed using the clustering, optimization and distance-based approaches. An optimization-based incremental clustering algorithm is proposed to find clusters of different types of cyberattacks. Using a special procedure, a set of clusters is divided into two subsets: normal and stable clusters. Then, outliers are found among stable clusters using an average distance between centroids of normal clusters. The proposed algorithm is evaluated using the well-known IDS data sets—Knowledge Discovery and Data mining Cup 1999 and UNSW-NB15—and compared with some other existing algorithms. Results show that the proposed algorithm has a high detection accuracy and its false negative rate is very low.
Similar content being viewed by others
References
Chen, P.T., Laih, C.S.: Idsic: an intrusion detection system with identification capability. Int. J. Inf. Secur. 7(3), 185–197 (2008)
Liao, H.J., Lin, Y.C., Lin, C.H.R., Tung, K.Y.: Review: intrusion detection system: a comprehensive review. J. Netw. Comput. Appl. 36(1), 16–24 (2013)
McHugh, J.: Intrusion and intrusion detection. Int. J. Inf. Secur. 1(1), 14–35 (2001)
Global information security practices: survey key findings and trends. http://www.pwc.com (2015). Accessed 2018
Sommer, R., Paxson, V.: Outside the closed world: On using machine learning for network intrusion detection. In: Security and Privacy, pp. 305–316 (2010)
Umer, M.F., Sher, M., Bi, X.: A two-stage flow-based intrusion detection model for next-generation networks. PloS One 13, e0180945 (2018)
Archana, D.W., Chatur, P.N.: Comparison of firewall and intrusion detection system. Int. J. Comput. Sci. Inf. Technol. 5(1), 674–678 (2014)
Kanika, U.: Security of network using Ids and firewall. Int. J. Sci. Res. Publ. 3(6), 1–4 (2013)
Chakir, E.M., Codjovi, C., Khamlichi, Y.I., Moughit, M., First Settat, H.: False positives reduction in intrusion detection systems using alert correlation and data mining techniques. Int. J. Adv. Res. Comput. Sci. Softw. Eng. IJARCSSE 5, 77–85 (2015)
Gupta, N., Srivastava, K., Sharma, A.: Reducing false positive in intrusion detection system: a survey. Int. J. Comput. Sci. Inf. Technol. 7, 1600–1603 (2016)
Lippmann, R.P., Fried, D.J., Graf, I., Haines, J.W., Kendall, K.R., Mcclung, D., et al.: Evaluating intrusion detection systems: the 1998 darpa off-line intrusion detection evaluation. DARPA Inf. Surviv. Conf. Expos. 2, 12–26 (2000)
Spathoulas, G.P., Katsikas, S.K.: Reducing false positives in intrusion detection systems. Comput. Secur. 29(1), 35–44 (2010)
Spathoulas, G.P., Katsikas, S.K.: Reducing false positives in intrusion detection systems. Comput. Secur. 29(1), 35–44 (2010)
Kadam, P.U., Deshmukh, M.: Various approaches for intrusion detection system: an overview. Int. J. Innov. Res. Comput. Commun. Eng. 2(11), 6894–6902 (2014)
Pareek, V., Mishra, A., Sharma, A., Chauhan, R., Bansal, S.: A deviation based outlier intrusion detection system. In: Chaki, N., Nagamalai, D., Meghanathan, N., Boumerdassi, S. (eds.) Recent Trends in Network Security and Applications, pp. 395–401. Springer, Berlin (2010)
Mujumdar, A., Masiwal, G.,Dr. Meshram, B.B.: Analysis of signature-based and behavior-based anti-malware approaches. Int. J. Adv. Res. Comput. Eng. Tech. (IJARCET). 2(6), 2037–2039 (2013)
Julisch, K.: Clustering intrusion detection alarms to support root cause analysis. ACM Trans. Inf. Syst. Secur. 6(4), 443–471 (2003)
Rubin, S., Jha, S., Miller, B.P.: Automatic generation and analysis of NIDS attacks. In: Proceedings of the 20th Annual Computer Security Applications Conference, pp. 28–38. IEEE Computer Society, Washington, DC (2004)
Mishra, P., Varadharajan, V., Tupakula, U., Pilli, E.S.: A detailed investigation and analysis of using machine learning techniques for intrusion detection. IEEE Commun. Surv. Tutor. 21(1), 686–728 (2019)
Li, Z., Das, A., Zhou, J.: Usaid: unifying signature-based and anomaly-based intrusion detection. In: Ho, T.B., Cheung, D., Liu, H. (eds.) Advances in Knowledge Discovery and Data Mining, pp. 702–712. Springer, Berlin (2005)
Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM Conference on Computer and Communications Security, pp. 255–264. ACM, New York (2002)
Breunig, M., Kriegel, H., Ng, R., Sander, J.: Lof: identifying density-based local outliers. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, pp. 93–104 (2000)
Schubert, E., Zimek, A., Kriegel, H.P.: Generalized Outlier Detection with Flexible Kernel Density Estimates, pp. 542–550 (2014)
Zhang, K., Hutter, M., Jin, H.: A new local distance-based outlier detection approach for scattered real-world data. In: Theeramunkong, T., Kijsirikul, B., Cercone, N., Ho, T.B. (eds.) Advances in Knowledge Discovery and Data Mining, pp. 813–822. Springer, Berlin (2009)
Zhu, Q., Feng, J., Huang, J.: Natural neighbor: a self-adaptive neighborhood method without parameter k. Pattern Recognit. Lett. 80, 30–36 (2016)
Jiang, M.F., Tseng, S.S., Su, C.M.: Two-phase clustering process for outliers detection. Pattern Recognit. Lett. 22(6), 691–700 (2001)
Wang, C.H.: Outlier identification and market segmentation using kernel based clustering techniques. Expert Syst. Appl. 36(2), 3744–3750 (2009)
Lian, D., Xu, L., Liu, Y., Lee, J.: Cluster-based outlier detection. Ann. Oper. Res. 168(1), 151–168 (2009)
Hachmi, F., Boujenfa, K., Limam, M.: An optimization process to identify outliers generated by intrusion detection systems. Secur. Commun. Netw. 8(18), 3469–3480 (2015)
Pachgade, S.D., Dhande, S.S.: A heuristic algorithm for solving the minimum sum-of-squares clustering problems. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 2(6), 12–16 (2012)
Rizk, H., ElGokhy, M., Sarhan, A.: A hybrid outlier detection algorithm based on partitioning clustering and density measures. In: 2015 Tenth International Conference on Computer Engineering and Systems (ICCES), pp. 175–181 (2015)
Dickson, A., Thomas, C.: Optimizing false alerts using multi-objective particle swarm optimization method. In: IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (2015)
Olsson, C., Eriksson, A., Hartley, R.: Outlier removal using duality. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2010)
Seo, Y., Lee, H., Lee, S.: Outlier removal by convex optimization for l-infinity approaches. In: Toshikazu, W., Fay, H., Stephen, L. (eds.) Advances in Image and Video Technology, pp. 203–214. Springer, Heidelberg (2009)
Cannady, J., Harrell, J.: A comparative analysis of current intrusion detection technologies. In: Proc. of the Fourth Technology for Information Security Conference’96 (TISC’96) (2000)
Bagirov, A.M., Ordin, B., Ozturk, G., Xavier, A.E.: An incremental clustering algorithm based on hyperbolic smoothing. Comput. Optim. Appl. 61(1), 219–241 (2015)
Bagirov, A.M., Taheri, S., Ugon, J.: Nonsmooth DC programming approach to the minimum sum-of-squares clustering problems. Pattern Recognit. 53, 12–24 (2016)
Ordin, B., Bagirov, A.M.: A heuristic algorithm for solving the minimum sum-of-squares clustering problems. J. Global Optim. 61(2), 341–361 (2015)
Bagirov, A.M.: Modified global k-means algorithm for minimum sum-of squares clustering problems. Pattern Recognit. 41(10), 3192–3199 (2008)
Madsen, J.H.: Distance and density-based outlier detection. https://github.com/jhmadsen/DDoutlier (2018). Accessed 2018
Network-intrusion-detection-using-machine-learning. https://github.com/Anshumank399/Network-Intrusion-Detection-using-Machine-Learning (2018). Accessed 2018
Dua, D., Graff, C.: UCI Machine Learning Repository, Irvine, CA: University of California, School of Information and Computer Sciences. http://archive.ics.uci.edu/ml (2019). Accessed 2018
Tan, P.N., Steinbach, M., Kumar, V.: Introduction to Data Mining. Addison-Wesley Longman Publishing Co., Inc., Boston (2005)
Nour, M., Jill, S.: Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In: Military Communications and Information Systems Conference, IEEE, pp. 1–6 (2015)
Acknowledgements
The authors are grateful to the anonymous reviewers for their constructive comments which greatly helped improving the quality of this paper.
Funding
This research was conducted in Internet Commerce Security Laboratory (ICSL) funded by Westpac Banking Corporation Australia. In addition, the research by Dr. Sona Taheri and A/Prof. Adil Bagirov was supported by the Australian Government through the Australian Research Council’s Discovery Projects funding scheme (DP190100580).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Taheri, S., Bagirov, A.M., Gondal, I. et al. Cyberattack triage using incremental clustering for intrusion detection systems. Int. J. Inf. Secur. 19, 597–607 (2020). https://doi.org/10.1007/s10207-019-00478-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10207-019-00478-3