Abstract
The introduction of clustering techniques enabled outlier detection on log lines independent from their syntax, thereby removing the need for parsers. However, clustering methods only produce static collections of clusters. Therefore, such approaches frequently require a reformation of the clusters in dynamic environments due to changes in technical infrastructure. Moreover, clustering alone is not able to detect anomalies that do not manifest themselves as outliers but rather as log lines with spurious frequencies or incorrect periodicity. In order to overcome these deficiencies, in this chapter we introduce a dynamic anomaly detection approach that generates multiple consecutive cluster maps and connects them by deploying cluster evolution techniques. For this, we design a novel clustering model that allows tracking clusters and determining their transitions. We detect anomalous system behavior by applying time-series analysis to relevant metrics computed from the evolving clusters.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Deepayan Chakrabarti, Ravi Kumar, and Andrew Tomkins. Evolutionary clustering. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 554–560, 2006.
Yun Chi, Xiaodan Song, Dengyong Zhou, Koji Hino, and Belle L Tseng. On evolutionary spectral clustering. ACM Transactions on Knowledge Discovery from Data (TKDD), 3(4):1–30, 2009.
J.D. Cryer and K.S. Chan. Time Series Analysis: With Applications in R. Springer Texts in Statistics. Springer, 2008.
R Gerhards. The syslog protocol: Rfc 5424. IETF Trust: Reston, VA, USA, 2009.
Derek Greene, Donal Doyle, and Padraig Cunningham. Tracking the evolution of communities in dynamic social networks. In 2010 international conference on advances in social networks analysis and mining, pages 176–183. IEEE, 2010.
Shilin He, Jieming Zhu, Pinjia He, and Michael R Lyu. Experience report: System log analysis for anomaly detection. In 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pages 207–218. IEEE, 2016.
Max Landauer, Markus Wurzenberger, Florian Skopik, Giuseppe Settanni, and Peter Filzmoser. Dynamic log file analysis: an unsupervised cluster evolution approach for anomaly detection. computers & security, 79:94–116, 2018.
Max Landauer, Markus Wurzenberger, Florian Skopik, Giuseppe Settanni, and Peter Filzmoser. Time series analysis: unsupervised anomaly detection beyond outlier detection. In International Conference on Information Security Practice and Experience, pages 19–36. Springer, 2018.
Myra Spiliopoulou, Irene Ntoutsi, Yannis Theodoridis, and Rene Schult. Monic: modeling and monitoring cluster transitions. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 706–711, 2006.
Masashi Toyoda and Masaru Kitsuregawa. Extracting evolution of web communities from a series of web archives. In Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, pages 28–37, 2003.
Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I Jordan. Detecting large-scale system problems by mining console logs. In Proceedings of the 22nd Symposium on Operating Systems Principles, pages 117–132. ACM, 2009.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Skopik, F., Wurzenberger, M., Landauer, M. (2021). Time Series Analysis for Temporal Anomaly Detection. In: Smart Log Data Analytics. Springer, Cham. https://doi.org/10.1007/978-3-030-74450-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-74450-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-74449-6
Online ISBN: 978-3-030-74450-2
eBook Packages: Computer ScienceComputer Science (R0)