Skip to main content
SpringerLink
Log in

New data mining technique to enhance IDS alarms quality

  • Original Paper
  • Published:
Journal in Computer Virology Aims and scope Submit manuscript

Abstract

The intrusion detection systems (IDSs) generate large number of alarms most of which are false positives. Fortunately, there are reasons for triggering alarms where most of these reasons are not attacks. In this paper, a new data mining technique has been developed to group alarms and to produce clusters. Hereafter, each cluster abstracted as a generalized alarm. The generalized alarms related to root causes are converted to filters to reduce future alarms load. The proposed algorithm makes use of nearest neighboring and generalization concepts to cluster alarms. As a clustering algorithm, the proposed algorithm uses a new measure to compute distances between alarms features values. This measure depends on background knowledge of the monitored network, making it robust and meaningful. The new data mining technique was verified with many datasets, and the averaged reduction ratio was about 82% of the total alarms. Application of the new technique to alarms log greatly helps the security analyst in identifying the root causes; and then reduces the alarm load in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Manganaris S., Christensen M., Zerkle D., Hermiz K.: A data mining analysis of RTID alarms. J. Comput. Netw. 34, 571–577 (2000)

    Article  Google Scholar 

  2. Julisch K.: Clustering intrusion detection alarms to support root cause analysis. ACM Trans. Inf. Syst. Secur. 6, 443–471 (2003)

    Article  Google Scholar 

  3. Yu J., Reddy Y.V.R., Selliah S., Reddy S., Bharadwaj V., Kankanahalli S.: TRINETR: an architecture for collaborative intrusion detection and knowledge-based alert evaluation. J. Adv. Eng. Inf. 19, 93–101 (2005)

    Article  Google Scholar 

  4. Perdisci R., Giacinto G., Roli F.: Alarm clustering for intrusion detection systems in computer networks. J. Eng. Appl. Artif. Intell. 19, 429–438 (2006)

    Article  Google Scholar 

  5. Siraj, A., Vaughn, R.: Multi-level alert clustering for intrusion detection sensor data. In: Proceeding of North American Fuzzy Information Processing Society International Conference on Soft Computing for Real World Applications, Michigan (2005)

  6. Julisch, K.: Using root cause analysis to handle intrusion detection alarms, PhD dissertation, University of Dortmund (2003)

  7. Al-Mamory, S.O., Zhang, H.: A survey on IDS alerts processing techniques. In: Proceeding of the 6th WSEAS International Conference on Information Security and Privacy (ISP ’07), Spain, pp. 69–78 (2007)

  8. Valdes, A., Skinner, K.: Probabilistic alert correlation. In: Proceeding of the Recent Advances in Intrusion Detection. LNCS, vol. 2212, pp. 54–68 (2001)

  9. Dain, O.M., Cunningham, R.K.: Fusing a heterogeneous alert stream into scenarios. In: Proceeding of the 2001 ACM Workshop on Data Mining for Security Applications, pp. 231–235 (2001)

  10. Ning P., Cui Y., Reeves D.S., Xu D.: Techniques and tools for analyzing intrusion alerts. ACM Trans. Inf. Syst. Secur. 7, 274–318 (2004)

    Article  Google Scholar 

  11. Morin, B., Me, L., Debar, H., Ducasse, M.: M2D2: A formal data model for IDS alert correlation. In: Proceeding of the International Symposium on Recent Advances in Intrusion Detection, pp. 115–137 (2002)

  12. Pietraszek, T.: Using adaptive alert classification to reduce false positives in intrusion detection. In: Proceeding of the Recent advances in intrusion detection, France, pp. 102–124 (2004)

  13. Hand D., Mannila H., Smyth P.: Principles of Data Mining. The MIT Press, Cambridge (2001)

    Google Scholar 

  14. Bellovin S.M.: Packets found on an Internet. J. Comput. Commun. Rev. 23, 26–31 (1993)

    Article  Google Scholar 

  15. Han J., Fu Y.: Exploration of the power of attribute-oriented induction in data mining. In: Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R.(eds) Advances in Knowledge Discovery and Data Mining, pp. 399–421. AAAI/MIT Press, Cambridge (1996)

    Google Scholar 

  16. Han J., Cai Y., Cercone N.: Data-driven discovery of quantitative rules in relational databases. IEEE Trans. Knowl. Data Eng. 5, 29–40 (1993)

    Article  Google Scholar 

  17. Julisch, K.: Mining alarm clusters to improve alarm handling efficiency. In: Proceeding of the 17th Annual Computer Security Applications Conference, New Orleans, pp. 12–21 (2001)

  18. Jain A.K., Murty M.N., Flynn P.J.: Data clustering: a review. ACM Comput. Surv. 31, 264–323 (1999)

    Article  Google Scholar 

  19. Theodoridis S., Koutroubas K.: Pattern Recognition. Academic Press, New York (1999)

    Google Scholar 

  20. Halkidi M., Batistakis Y., Vazirgiannis M.: On clustering validation techniques. J. Intell. Inf. Syst. 17, 107–145 (2001)

    Article  MATH  Google Scholar 

  21. Berry M.J.A., Linoff G.: Data Mining Techniques for Marketing, Sales and Customer Support. Wiley, New York (1996)

    Google Scholar 

  22. Halkidi, M., Vazirgiannis, M., Batistakis, I.: Quality scheme assessment in the clustering process. In: Proceeding of the 4th European Conference on Principles of Data Mining and Knowledge Discovery, pp. 265–276 (2000)

  23. Rezaee M.R., Lelieveldt B.B.F., Reiber J.H.C.: A new cluster validity index for the fuzzy c-mean. Pattern Recognit. Lett. 19, 237–246 (1998)

    Article  MATH  Google Scholar 

  24. Sharma S.C.: Applied Multivariate Techniques. Wiley, New York (1996)

    Google Scholar 

  25. Xie X.L., Beni G.: A validity measure for fuzzy clustering. IEEE Trans. Pattern Anal. Mach. Intell. 13(8), 841–847 (1991)

    Article  Google Scholar 

  26. Rousseeuw P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20(1), 53–65 (1987)

    Article  MATH  Google Scholar 

  27. Bezdek J.C., Pal N.R.: Some new indexes of cluster validity. IEEE Trans. Syst. Man Cybern. Part B 28(3), 301–315 (1998)

    Article  Google Scholar 

  28. Davies D.L., Bouldin D.W.: A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell. 1(2), 224–227 (1979)

    Article  Google Scholar 

  29. MIT Lincoln Laboratory: 1999 DARPA intrusion detection evaluation data set (1999). http://www.ll.mit.edu/IST/ideval/data/1999/1999dataindex.html

  30. Roesch, M.: Snort-lightweight intrusion detection for networks. In: Proceeding of the 1999 USENIX LISA Conference, pp. 229–238 (1999)

  31. Valeur, F., Vigna, G., Kruegel, C., Kemmerer, R.A.: A comprehensive approach to intrusion detection alert correlation. In: IEEE Transactions on Dependable and Secure Computing 1(3) (2004)

  32. Pietraszek, T.: Alert classification to reduce false positives in intrusion detection. PhD dissertation, Institut für Informatik, Albert-Ludwigs-Universität Freiburg, Germany, July 2006

  33. Wang, J., Lee, I.: Measuring false-positive by automated real-time correlated hacking behavior analysis. In: Proceedings of the 4th International Conference on Information Security. LNCS, vol. 2200, pp. 512–535 (2001)

  34. Valeur, F.: Real-time intrusion detection alert correlation. PhD dissertation, University of California, Santa Barbara, June 2006

  35. Han, J., Cai, Y., Cercone, N.: Knowledge discovery in databases: an attribute-oriented approach. In: Proceeding of the 18th International Conference on Very Large Databases, pp. 547–559 (1992)

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, 150001, Harbin, China

    Safaa O. Al-Mamory & Hongli Zhang

Authors
  1. Safaa O. Al-Mamory
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongli Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Safaa O. Al-Mamory.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Al-Mamory, S.O., Zhang, H. New data mining technique to enhance IDS alarms quality. J Comput Virol 6, 43–55 (2010). https://doi.org/10.1007/s11416-008-0104-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11416-008-0104-2

Keywords

Search

Navigation