Abstract
The intrusion detection systems (IDSs) generate large number of alarms most of which are false positives. Fortunately, there are reasons for triggering alarms where most of these reasons are not attacks. In this paper, a new data mining technique has been developed to group alarms and to produce clusters. Hereafter, each cluster abstracted as a generalized alarm. The generalized alarms related to root causes are converted to filters to reduce future alarms load. The proposed algorithm makes use of nearest neighboring and generalization concepts to cluster alarms. As a clustering algorithm, the proposed algorithm uses a new measure to compute distances between alarms features values. This measure depends on background knowledge of the monitored network, making it robust and meaningful. The new data mining technique was verified with many datasets, and the averaged reduction ratio was about 82% of the total alarms. Application of the new technique to alarms log greatly helps the security analyst in identifying the root causes; and then reduces the alarm load in the future.
Similar content being viewed by others
References
Manganaris S., Christensen M., Zerkle D., Hermiz K.: A data mining analysis of RTID alarms. J. Comput. Netw. 34, 571–577 (2000)
Julisch K.: Clustering intrusion detection alarms to support root cause analysis. ACM Trans. Inf. Syst. Secur. 6, 443–471 (2003)
Yu J., Reddy Y.V.R., Selliah S., Reddy S., Bharadwaj V., Kankanahalli S.: TRINETR: an architecture for collaborative intrusion detection and knowledge-based alert evaluation. J. Adv. Eng. Inf. 19, 93–101 (2005)
Perdisci R., Giacinto G., Roli F.: Alarm clustering for intrusion detection systems in computer networks. J. Eng. Appl. Artif. Intell. 19, 429–438 (2006)
Siraj, A., Vaughn, R.: Multi-level alert clustering for intrusion detection sensor data. In: Proceeding of North American Fuzzy Information Processing Society International Conference on Soft Computing for Real World Applications, Michigan (2005)
Julisch, K.: Using root cause analysis to handle intrusion detection alarms, PhD dissertation, University of Dortmund (2003)
Al-Mamory, S.O., Zhang, H.: A survey on IDS alerts processing techniques. In: Proceeding of the 6th WSEAS International Conference on Information Security and Privacy (ISP ’07), Spain, pp. 69–78 (2007)
Valdes, A., Skinner, K.: Probabilistic alert correlation. In: Proceeding of the Recent Advances in Intrusion Detection. LNCS, vol. 2212, pp. 54–68 (2001)
Dain, O.M., Cunningham, R.K.: Fusing a heterogeneous alert stream into scenarios. In: Proceeding of the 2001 ACM Workshop on Data Mining for Security Applications, pp. 231–235 (2001)
Ning P., Cui Y., Reeves D.S., Xu D.: Techniques and tools for analyzing intrusion alerts. ACM Trans. Inf. Syst. Secur. 7, 274–318 (2004)
Morin, B., Me, L., Debar, H., Ducasse, M.: M2D2: A formal data model for IDS alert correlation. In: Proceeding of the International Symposium on Recent Advances in Intrusion Detection, pp. 115–137 (2002)
Pietraszek, T.: Using adaptive alert classification to reduce false positives in intrusion detection. In: Proceeding of the Recent advances in intrusion detection, France, pp. 102–124 (2004)
Hand D., Mannila H., Smyth P.: Principles of Data Mining. The MIT Press, Cambridge (2001)
Bellovin S.M.: Packets found on an Internet. J. Comput. Commun. Rev. 23, 26–31 (1993)
Han J., Fu Y.: Exploration of the power of attribute-oriented induction in data mining. In: Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R.(eds) Advances in Knowledge Discovery and Data Mining, pp. 399–421. AAAI/MIT Press, Cambridge (1996)
Han J., Cai Y., Cercone N.: Data-driven discovery of quantitative rules in relational databases. IEEE Trans. Knowl. Data Eng. 5, 29–40 (1993)
Julisch, K.: Mining alarm clusters to improve alarm handling efficiency. In: Proceeding of the 17th Annual Computer Security Applications Conference, New Orleans, pp. 12–21 (2001)
Jain A.K., Murty M.N., Flynn P.J.: Data clustering: a review. ACM Comput. Surv. 31, 264–323 (1999)
Theodoridis S., Koutroubas K.: Pattern Recognition. Academic Press, New York (1999)
Halkidi M., Batistakis Y., Vazirgiannis M.: On clustering validation techniques. J. Intell. Inf. Syst. 17, 107–145 (2001)
Berry M.J.A., Linoff G.: Data Mining Techniques for Marketing, Sales and Customer Support. Wiley, New York (1996)
Halkidi, M., Vazirgiannis, M., Batistakis, I.: Quality scheme assessment in the clustering process. In: Proceeding of the 4th European Conference on Principles of Data Mining and Knowledge Discovery, pp. 265–276 (2000)
Rezaee M.R., Lelieveldt B.B.F., Reiber J.H.C.: A new cluster validity index for the fuzzy c-mean. Pattern Recognit. Lett. 19, 237–246 (1998)
Sharma S.C.: Applied Multivariate Techniques. Wiley, New York (1996)
Xie X.L., Beni G.: A validity measure for fuzzy clustering. IEEE Trans. Pattern Anal. Mach. Intell. 13(8), 841–847 (1991)
Rousseeuw P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20(1), 53–65 (1987)
Bezdek J.C., Pal N.R.: Some new indexes of cluster validity. IEEE Trans. Syst. Man Cybern. Part B 28(3), 301–315 (1998)
Davies D.L., Bouldin D.W.: A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell. 1(2), 224–227 (1979)
MIT Lincoln Laboratory: 1999 DARPA intrusion detection evaluation data set (1999). http://www.ll.mit.edu/IST/ideval/data/1999/1999dataindex.html
Roesch, M.: Snort-lightweight intrusion detection for networks. In: Proceeding of the 1999 USENIX LISA Conference, pp. 229–238 (1999)
Valeur, F., Vigna, G., Kruegel, C., Kemmerer, R.A.: A comprehensive approach to intrusion detection alert correlation. In: IEEE Transactions on Dependable and Secure Computing 1(3) (2004)
Pietraszek, T.: Alert classification to reduce false positives in intrusion detection. PhD dissertation, Institut für Informatik, Albert-Ludwigs-Universität Freiburg, Germany, July 2006
Wang, J., Lee, I.: Measuring false-positive by automated real-time correlated hacking behavior analysis. In: Proceedings of the 4th International Conference on Information Security. LNCS, vol. 2200, pp. 512–535 (2001)
Valeur, F.: Real-time intrusion detection alert correlation. PhD dissertation, University of California, Santa Barbara, June 2006
Han, J., Cai, Y., Cercone, N.: Knowledge discovery in databases: an attribute-oriented approach. In: Proceeding of the 18th International Conference on Very Large Databases, pp. 547–559 (1992)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Al-Mamory, S.O., Zhang, H. New data mining technique to enhance IDS alarms quality. J Comput Virol 6, 43–55 (2010). https://doi.org/10.1007/s11416-008-0104-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11416-008-0104-2