Effective Dimension in Anomaly Detection: Its Application to Computer Systems
Conference paper
- 423 Downloads
Abstract
We consider the issue of online anomaly detection from a time sequence of directional data (normalized vectors) in high dimensional systems. In spite of the practical importance, little is known about anomaly detection methods for directional data. Using a novel concept of the effective dimension of the system, we successfully formulated an anomaly detection method which is free from the “curse of dimensionality.” In our method, we derive a probability distribution function (pdf) for an anomaly metric, and use a novel update algorithm for the parameters in the pdf, where the effective dimension is included as a fitting parameter. For directional data from a computer system, we demonstrate the utility of our algorithm in anomaly detection.
Keywords
Singular Value Decomposition Anomaly Detection Dependency Graph Online Algorithm Directional Data
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Preview
Unable to display preview. Download preview PDF.
References
- 1.Banerjee, A., Dhillon, I., Ghosh, J., Sra, S.: Expectation maximization for clustering on hyperspheres. Technical Report, TR-03-07, Department of Computer Sciences, University of Texas at Austin (2003)Google Scholar
- 2.Banerjee, A., Dhillon, I., Ghosh, J., Sra, S.: Generative model-based clustering of directional data. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 19–28. ACM Press, New York (2003)CrossRefGoogle Scholar
- 3.Berman, A., Plemmons, R.J.: Nonnegative Matrices in the Mathematical Sciences. Classics in applied mathematics, vol. 9. SIAM, Philadelphia (1994)zbMATHGoogle Scholar
- 4.Deerwester, S.C., Dumais, S.T., Landauer, T.K., Furnas, G.W., Harshman, R.A.: Indexing by latent semantic analysis. Journal of the American Society of Information Science 41(6), 391–407 (1990)CrossRefGoogle Scholar
- 5.Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wiley, Chichester (2000)Google Scholar
- 6.Gupta, M., Neogi, A., Agarwal, M.K., Kar, G.: Discovering dynamic dependencies in enterprise environments for problem determination. In: Proceedings of 14th IFIP/IEEE Workshop on Distributed Systems: Operations and Management, pp. 221–233. IEEE Computer Society Press, Los Alamitos (2003)Google Scholar
- 7.
- 8.Idé, T., Kashima, H.: Eigenspace-based anomaly detection in computer systems. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM Press, New York (2004)Google Scholar
- 9.Jaakkola, T., Haussler, D.: Exploiting generative models in discriminative classifiers. Advances in Neural Information Processing Systems 11, 487–493 (1999)Google Scholar
- 10.Mardia, K.V.: Multivariate Analysis. Academic Press, London (1980)Google Scholar
- 11.Sarkar, S., Boyer, K.: Quantitative measures for change based on feature organization: Eigenvalues and eigenvectors. Computer Vision and Image Understanding 71, 110–136 (1998)CrossRefGoogle Scholar
- 12.Strang, G.: Linear Algebra and its Applications. Academic Press, London (1976)zbMATHGoogle Scholar
- 13.The Open Group. Application response measurement — ARM. http://www.opengroup.org/tech/management/arm/
- 14.Yamanishi, K., Takeuchi, J.: A unifying framework for detecting outliers and change points from non-stationary time series data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 676–681. ACM Press, New York (2002)CrossRefGoogle Scholar
- 15.Yamanishi, K., Takeuchi, J., Williams, G., Milne, P.: On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 320–324. ACM Press, New York (2000)CrossRefGoogle Scholar
Copyright information
© Springer Berlin Heidelberg 2007