Effective Dimension in Anomaly Detection: Its Application to Computer Systems

  • Tsuyoshi Idé
  • Hisashi Kashima
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3609)

Abstract

We consider the issue of online anomaly detection from a time sequence of directional data (normalized vectors) in high dimensional systems. In spite of the practical importance, little is known about anomaly detection methods for directional data. Using a novel concept of the effective dimension of the system, we successfully formulated an anomaly detection method which is free from the “curse of dimensionality.” In our method, we derive a probability distribution function (pdf) for an anomaly metric, and use a novel update algorithm for the parameters in the pdf, where the effective dimension is included as a fitting parameter. For directional data from a computer system, we demonstrate the utility of our algorithm in anomaly detection.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Banerjee, A., Dhillon, I., Ghosh, J., Sra, S.: Expectation maximization for clustering on hyperspheres. Technical Report, TR-03-07, Department of Computer Sciences, University of Texas at Austin (2003)Google Scholar
  2. 2.
    Banerjee, A., Dhillon, I., Ghosh, J., Sra, S.: Generative model-based clustering of directional data. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 19–28. ACM Press, New York (2003)CrossRefGoogle Scholar
  3. 3.
    Berman, A., Plemmons, R.J.: Nonnegative Matrices in the Mathematical Sciences. Classics in applied mathematics, vol. 9. SIAM, Philadelphia (1994)MATHGoogle Scholar
  4. 4.
    Deerwester, S.C., Dumais, S.T., Landauer, T.K., Furnas, G.W., Harshman, R.A.: Indexing by latent semantic analysis. Journal of the American Society of Information Science 41(6), 391–407 (1990)CrossRefGoogle Scholar
  5. 5.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wiley, Chichester (2000)Google Scholar
  6. 6.
    Gupta, M., Neogi, A., Agarwal, M.K., Kar, G.: Discovering dynamic dependencies in enterprise environments for problem determination. In: Proceedings of 14th IFIP/IEEE Workshop on Distributed Systems: Operations and Management, pp. 221–233. IEEE Computer Society Press, Los Alamitos (2003)Google Scholar
  7. 7.
  8. 8.
    Idé, T., Kashima, H.: Eigenspace-based anomaly detection in computer systems. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM Press, New York (2004)Google Scholar
  9. 9.
    Jaakkola, T., Haussler, D.: Exploiting generative models in discriminative classifiers. Advances in Neural Information Processing Systems 11, 487–493 (1999)Google Scholar
  10. 10.
    Mardia, K.V.: Multivariate Analysis. Academic Press, London (1980)Google Scholar
  11. 11.
    Sarkar, S., Boyer, K.: Quantitative measures for change based on feature organization: Eigenvalues and eigenvectors. Computer Vision and Image Understanding 71, 110–136 (1998)CrossRefGoogle Scholar
  12. 12.
    Strang, G.: Linear Algebra and its Applications. Academic Press, London (1976)MATHGoogle Scholar
  13. 13.
    The Open Group. Application response measurement — ARM. http://www.opengroup.org/tech/management/arm/
  14. 14.
    Yamanishi, K., Takeuchi, J.: A unifying framework for detecting outliers and change points from non-stationary time series data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 676–681. ACM Press, New York (2002)CrossRefGoogle Scholar
  15. 15.
    Yamanishi, K., Takeuchi, J., Williams, G., Milne, P.: On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 320–324. ACM Press, New York (2000)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Tsuyoshi Idé
    • 1
  • Hisashi Kashima
    • 1
  1. 1.IBM Research, Tokyo Research Laboratory, 1623-14 Shimotsuruma, Yamato-shi, Kanagawa 242-8502Japan

Personalised recommendations