Skip to main content

A new data normalization method for unsupervised anomaly intrusion detection

Abstract

Unsupervised anomaly detection can detect attacks without the need for clean or labeled training data. This paper studies the application of clustering to unsupervised anomaly detection (ACUAD). Data records are mapped to a feature space. Anomalies are detected by determining which points lie in the sparse regions of the feature space. A critical element for this method to be effective is the definition of the distance function between data records. We propose a unified normalization distance framework for records with numeric and nominal features mixed data. A heuristic method that computes the distance for nominal features is proposed, taking advantage of an important characteristic of nominal features—their probability distribution. Then, robust methods are proposed for mapping numeric features and computing their distance, these being able to tolerate the impact of the value difference in scale and diversification among features, and outliers introduced by intrusions. Empirical experiments with the KDD 1999 dataset showed that ACUAD can detect intrusions with relatively low false alarm rates compared with other approaches.

This is a preview of subscription content, access via your institution.

References

  • Cansado, A., Soto, A., 2008. Unsupervised anomaly detection in large databases using Bayesian networks. Appl. Artif. Intell., 22(4):309–330. [doi:10.1080/08839510801972801]

    Article  Google Scholar 

  • Eskin, E., 2000. Anomaly Detection over Noisy Data Using Learned Probability Distributions. Proc. Int. Conf. on Machine Learning, p.255–262. [doi:10.1109/ICCSA.2008.70]

  • Eskin, E., Arnold, A., Prerau, M., Portony, L., Stolfo, S., 2002. A Geometric Framework for Unsupervised Anomaly Detection: Detecting Intrusions in Unlabeled Data. In: Barbara, E., Jajodia, S. (Eds.), Applications of Data Mining in Computer Security. Kluwer Academic Publishers, Norwell, MA, USA, p.272.

    Google Scholar 

  • Ismail, A.S.H., Abdullah, A.H., Bak, K.B.A., Nqudi, M.A., Dahlan, D., Chimphlee, W., 2008. A Novel Method for Unsupervised Anomaly Detection Using Unlabelled Data. Proc. Int. Conf. on Computational Sciences and Its Applications., p.252–260. [doi:10.1109/ICCSA.2008.70]

  • Knorr, E.M., 2002. Outliers and Data Mining: Finding Exceptions in Data. PhD Thesis, University of British Columbia, Canada, p.74.

    Google Scholar 

  • Kwitt, R., Hofmann, U., 2007. Unsupervised Anomaly Detection in Network Traffic by Means of Robust PCA. Proc. Int. Multi-Conf. on Computing in the Global Information Technology, p.37–41. [doi:10.1109/ICCGI.2007.62]

  • Leung, K., Leckie, C., 2005. Unsupervised Anomaly Detection in Network Intrusion Detection Using Clusters. Proc. 28th Australasian Conf. on Computer Science, 102:333–342.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Long-zheng Cai.

Additional information

Project supported by the PhD Foundation of Engineering and Commerce College, South-Central University for Nationalities, China

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Cai, Lz., Chen, J., Ke, Y. et al. A new data normalization method for unsupervised anomaly intrusion detection. J. Zhejiang Univ. - Sci. C 11, 778–784 (2010). https://doi.org/10.1631/jzus.C0910625

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1631/jzus.C0910625

Key words

  • Unsupervised anomaly detection
  • Data mining
  • Intrusion detection
  • Network security

CLC number

  • TP393.08