Skip to main content
SpringerLink
Log in

Cyberattack triage using incremental clustering for intrusion detection systems

  • Regular Contribution
  • Published:
International Journal of Information Security Aims and scope Submit manuscript

Abstract

Intrusion detection systems (IDSs) are devices or software applications that monitor networks or systems for malicious activities and signals alerts/alarms when such activity is discovered. However, an IDS may generate many false alerts which affect its accuracy. In this paper, we develop a cyberattack triage algorithm to detect these alerts (so-called outliers). The proposed algorithm is designed using the clustering, optimization and distance-based approaches. An optimization-based incremental clustering algorithm is proposed to find clusters of different types of cyberattacks. Using a special procedure, a set of clusters is divided into two subsets: normal and stable clusters. Then, outliers are found among stable clusters using an average distance between centroids of normal clusters. The proposed algorithm is evaluated using the well-known IDS data sets—Knowledge Discovery and Data mining Cup 1999 and UNSW-NB15—and compared with some other existing algorithms. Results show that the proposed algorithm has a high detection accuracy and its false negative rate is very low.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Chen, P.T., Laih, C.S.: Idsic: an intrusion detection system with identification capability. Int. J. Inf. Secur. 7(3), 185–197 (2008)

    Google Scholar 

  2. Liao, H.J., Lin, Y.C., Lin, C.H.R., Tung, K.Y.: Review: intrusion detection system: a comprehensive review. J. Netw. Comput. Appl. 36(1), 16–24 (2013)

    Google Scholar 

  3. McHugh, J.: Intrusion and intrusion detection. Int. J. Inf. Secur. 1(1), 14–35 (2001)

    MATH  Google Scholar 

  4. Global information security practices: survey key findings and trends. http://www.pwc.com (2015). Accessed 2018

  5. Sommer, R., Paxson, V.: Outside the closed world: On using machine learning for network intrusion detection. In: Security and Privacy, pp. 305–316 (2010)

  6. Umer, M.F., Sher, M., Bi, X.: A two-stage flow-based intrusion detection model for next-generation networks. PloS One 13, e0180945 (2018)

    Google Scholar 

  7. Archana, D.W., Chatur, P.N.: Comparison of firewall and intrusion detection system. Int. J. Comput. Sci. Inf. Technol. 5(1), 674–678 (2014)

    Google Scholar 

  8. Kanika, U.: Security of network using Ids and firewall. Int. J. Sci. Res. Publ. 3(6), 1–4 (2013)

    Google Scholar 

  9. Chakir, E.M., Codjovi, C., Khamlichi, Y.I., Moughit, M., First Settat, H.: False positives reduction in intrusion detection systems using alert correlation and data mining techniques. Int. J. Adv. Res. Comput. Sci. Softw. Eng. IJARCSSE 5, 77–85 (2015)

    Google Scholar 

  10. Gupta, N., Srivastava, K., Sharma, A.: Reducing false positive in intrusion detection system: a survey. Int. J. Comput. Sci. Inf. Technol. 7, 1600–1603 (2016)

    Google Scholar 

  11. Lippmann, R.P., Fried, D.J., Graf, I., Haines, J.W., Kendall, K.R., Mcclung, D., et al.: Evaluating intrusion detection systems: the 1998 darpa off-line intrusion detection evaluation. DARPA Inf. Surviv. Conf. Expos. 2, 12–26 (2000)

    Google Scholar 

  12. Spathoulas, G.P., Katsikas, S.K.: Reducing false positives in intrusion detection systems. Comput. Secur. 29(1), 35–44 (2010)

    Google Scholar 

  13. Spathoulas, G.P., Katsikas, S.K.: Reducing false positives in intrusion detection systems. Comput. Secur. 29(1), 35–44 (2010)

    Google Scholar 

  14. Kadam, P.U., Deshmukh, M.: Various approaches for intrusion detection system: an overview. Int. J. Innov. Res. Comput. Commun. Eng. 2(11), 6894–6902 (2014)

    Google Scholar 

  15. Pareek, V., Mishra, A., Sharma, A., Chauhan, R., Bansal, S.: A deviation based outlier intrusion detection system. In: Chaki, N., Nagamalai, D., Meghanathan, N., Boumerdassi, S. (eds.) Recent Trends in Network Security and Applications, pp. 395–401. Springer, Berlin (2010)

    Google Scholar 

  16. Mujumdar, A., Masiwal, G.,Dr. Meshram, B.B.: Analysis of signature-based and behavior-based anti-malware approaches. Int. J. Adv. Res. Comput. Eng. Tech. (IJARCET). 2(6), 2037–2039 (2013)

  17. Julisch, K.: Clustering intrusion detection alarms to support root cause analysis. ACM Trans. Inf. Syst. Secur. 6(4), 443–471 (2003)

    Google Scholar 

  18. Rubin, S., Jha, S., Miller, B.P.: Automatic generation and analysis of NIDS attacks. In: Proceedings of the 20th Annual Computer Security Applications Conference, pp. 28–38. IEEE Computer Society, Washington, DC (2004)

  19. Mishra, P., Varadharajan, V., Tupakula, U., Pilli, E.S.: A detailed investigation and analysis of using machine learning techniques for intrusion detection. IEEE Commun. Surv. Tutor. 21(1), 686–728 (2019)

    Google Scholar 

  20. Li, Z., Das, A., Zhou, J.: Usaid: unifying signature-based and anomaly-based intrusion detection. In: Ho, T.B., Cheung, D., Liu, H. (eds.) Advances in Knowledge Discovery and Data Mining, pp. 702–712. Springer, Berlin (2005)

    Google Scholar 

  21. Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM Conference on Computer and Communications Security, pp. 255–264. ACM, New York (2002)

  22. Breunig, M., Kriegel, H., Ng, R., Sander, J.: Lof: identifying density-based local outliers. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, pp. 93–104 (2000)

  23. Schubert, E., Zimek, A., Kriegel, H.P.: Generalized Outlier Detection with Flexible Kernel Density Estimates, pp. 542–550 (2014)

  24. Zhang, K., Hutter, M., Jin, H.: A new local distance-based outlier detection approach for scattered real-world data. In: Theeramunkong, T., Kijsirikul, B., Cercone, N., Ho, T.B. (eds.) Advances in Knowledge Discovery and Data Mining, pp. 813–822. Springer, Berlin (2009)

    Google Scholar 

  25. Zhu, Q., Feng, J., Huang, J.: Natural neighbor: a self-adaptive neighborhood method without parameter k. Pattern Recognit. Lett. 80, 30–36 (2016)

    Google Scholar 

  26. Jiang, M.F., Tseng, S.S., Su, C.M.: Two-phase clustering process for outliers detection. Pattern Recognit. Lett. 22(6), 691–700 (2001)

    MATH  Google Scholar 

  27. Wang, C.H.: Outlier identification and market segmentation using kernel based clustering techniques. Expert Syst. Appl. 36(2), 3744–3750 (2009)

    Google Scholar 

  28. Lian, D., Xu, L., Liu, Y., Lee, J.: Cluster-based outlier detection. Ann. Oper. Res. 168(1), 151–168 (2009)

    MathSciNet  MATH  Google Scholar 

  29. Hachmi, F., Boujenfa, K., Limam, M.: An optimization process to identify outliers generated by intrusion detection systems. Secur. Commun. Netw. 8(18), 3469–3480 (2015)

    Google Scholar 

  30. Pachgade, S.D., Dhande, S.S.: A heuristic algorithm for solving the minimum sum-of-squares clustering problems. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 2(6), 12–16 (2012)

    Google Scholar 

  31. Rizk, H., ElGokhy, M., Sarhan, A.: A hybrid outlier detection algorithm based on partitioning clustering and density measures. In: 2015 Tenth International Conference on Computer Engineering and Systems (ICCES), pp. 175–181 (2015)

  32. Dickson, A., Thomas, C.: Optimizing false alerts using multi-objective particle swarm optimization method. In: IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (2015)

  33. Olsson, C., Eriksson, A., Hartley, R.: Outlier removal using duality. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2010)

  34. Seo, Y., Lee, H., Lee, S.: Outlier removal by convex optimization for l-infinity approaches. In: Toshikazu, W., Fay, H., Stephen, L. (eds.) Advances in Image and Video Technology, pp. 203–214. Springer, Heidelberg (2009)

    Google Scholar 

  35. Cannady, J., Harrell, J.: A comparative analysis of current intrusion detection technologies. In: Proc. of the Fourth Technology for Information Security Conference’96 (TISC’96) (2000)

  36. Bagirov, A.M., Ordin, B., Ozturk, G., Xavier, A.E.: An incremental clustering algorithm based on hyperbolic smoothing. Comput. Optim. Appl. 61(1), 219–241 (2015)

    MathSciNet  MATH  Google Scholar 

  37. Bagirov, A.M., Taheri, S., Ugon, J.: Nonsmooth DC programming approach to the minimum sum-of-squares clustering problems. Pattern Recognit. 53, 12–24 (2016)

    MATH  Google Scholar 

  38. Ordin, B., Bagirov, A.M.: A heuristic algorithm for solving the minimum sum-of-squares clustering problems. J. Global Optim. 61(2), 341–361 (2015)

    MathSciNet  MATH  Google Scholar 

  39. Bagirov, A.M.: Modified global k-means algorithm for minimum sum-of squares clustering problems. Pattern Recognit. 41(10), 3192–3199 (2008)

    MATH  Google Scholar 

  40. Madsen, J.H.: Distance and density-based outlier detection. https://github.com/jhmadsen/DDoutlier (2018). Accessed 2018

  41. Network-intrusion-detection-using-machine-learning. https://github.com/Anshumank399/Network-Intrusion-Detection-using-Machine-Learning (2018). Accessed 2018

  42. Dua, D., Graff, C.: UCI Machine Learning Repository, Irvine, CA: University of California, School of Information and Computer Sciences. http://archive.ics.uci.edu/ml (2019). Accessed 2018

  43. Tan, P.N., Steinbach, M., Kumar, V.: Introduction to Data Mining. Addison-Wesley Longman Publishing Co., Inc., Boston (2005)

    Google Scholar 

  44. Nour, M., Jill, S.: Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In: Military Communications and Information Systems Conference, IEEE, pp. 1–6 (2015)

Download references

Acknowledgements

The authors are grateful to the anonymous reviewers for their constructive comments which greatly helped improving the quality of this paper.

Funding

This research was conducted in Internet Commerce Security Laboratory (ICSL) funded by Westpac Banking Corporation Australia. In addition, the research by Dr. Sona Taheri and A/Prof. Adil Bagirov was supported by the Australian Government through the Australian Research Council’s Discovery Projects funding scheme (DP190100580).

Author information

Authors and Affiliations

  1. Internet Commerce Security Laboratory, School of Science, Engineering and Information Technology, Federation University Australia, Ballarat, Australia

    Sona Taheri, Adil M. Bagirov & Iqbal Gondal

  2. Westpac Banking Corporation Australia, Sydney, Australia

    Simon Brown

Authors
  1. Sona Taheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adil M. Bagirov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Iqbal Gondal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Simon Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sona Taheri.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Taheri, S., Bagirov, A.M., Gondal, I. et al. Cyberattack triage using incremental clustering for intrusion detection systems. Int. J. Inf. Secur. 19, 597–607 (2020). https://doi.org/10.1007/s10207-019-00478-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10207-019-00478-3

Keywords

Search

Navigation