Abstract
Anomaly detection on log data is an important security mechanism that allows the detection of unknown attacks. Self-learning algorithms capture the behavior of a system over time and are able to identify deviations from the learned normal behavior online. The introduction of clustering techniques enabled outlier detection on log lines independent from their syntax, thereby removing the need for parsers. However, clustering methods only produce static collections of clusters. Therefore, such approaches frequently require a reformation of the clusters in dynamic environments due to changes in technical infrastructure. Moreover, clustering alone is not able to detect anomalies that do not manifest themselves as outliers but rather as log lines with spurious frequencies or incorrect periodicity. In order to overcome these deficiencies, in this paper we introduce a dynamic anomaly detection approach that generates multiple consecutive cluster maps and connects them by deploying cluster evolution techniques. For this, we design a novel clustering model that allows tracking clusters and determining their transitions. We detect anomalous system behavior by applying time-series analysis to relevant metrics computed from the evolving clusters. Finally, we evaluate our solution on an illustrative scenario and validate the achieved quality of the retrieved anomalies with respect to the runtime.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
References
Biggio, B., et al.: Poisoning behavioral malware clustering. In: Proceedings of the 2014 Workshop on Artificial Intelligent and Security Workshop, pp. 27–36. ACM (2014)
Chakrabarti, D., Kumar, R., Tomkins, A.: Evolutionary clustering. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 554–560. ACM (2006)
Chan, J., Bailey, J., Leckie, C.: Discovering correlated spatio-temporal changes in evolving graphs. Knowl. Inf. Syst. 16(1), 53–96 (2008)
Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 15 (2009)
Chi, Y., Song, X., Zhou, D., Hino, K., Tseng, B.L.: On evolutionary spectral clustering. ACM Trans. Knowl. Discov. Data (TKDD) 3(4), 17 p. (2009)
Cryer, J., Chan, K.: Time Series Analysis: With Applications in R. Springer Texts in Statistics. Springer, Heidelberg (2008). https://doi.org/10.1007/978-0-387-75959-3. https://books.google.at/books?id=MrNY3s2difIC
Greene, D., Doyle, D., Cunningham, P.: Tracking the evolution of communities in dynamic social networks. In: Advances in Social Networks Analysis and Mining (ASONAM), pp. 176–183. IEEE (2010)
He, S., Zhu, J., He, P., Lyu, M.R.: Experience report: system log analysis for anomaly detection. In: 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pp. 207–218. IEEE (2016)
Jensen, C.S., Lin, D., Ooi, B.C.: Continuous clustering of moving objects. IEEE Trans. Knowl. Data Eng. 19(9), 1161–1174 (2007)
Killick, R., Fearnhead, P., Eckley, I.A.: Optimal detection of changepoints with a linear computational cost. J. Am. Stat. Assoc. 107(500), 1590–1598 (2012)
Lughofer, E., Sayed-Mouchaweh, M.: Autonomous data stream clustering implementing split-and-merge concepts-towards a plug-and-play approach. Inf. Sci. 304, 54–79 (2015)
Pincombe, B.: Anomaly detection in time series of graphs using arma processes. Asor Bull. 24(4), 2 (2005)
Scarfone, K., Mell, P.: Guide to intrusion detection and prevention systems (IDPS). NIST Special Publication 800-94 (2007)
Skopik, F., Settanni, G., Fiedler, R., Friedberg, I.: Semi-synthetic data set generation for security software evaluation. In: 2014 Twelfth Annual International Conference on Privacy, Security and Trust (PST), pp. 156–163. IEEE (2014)
Spiliopoulou, M., Ntoutsi, I., Theodoridis, Y., Schult, R.: MONIC: modeling and monitoring cluster transitions. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 706–711. ACM (2006)
Toyoda, M., Kitsuregawa, M.: Extracting evolution of web communities from a series of web archives. In: Proceedings of the Fourteenth ACM Conference on Hypertext and Hypermedia, pp. 28–37. ACM (2003)
Wurzenberger, M., Skopik, F., Landauer, M., Greitbauer, P., Fiedler, R., Kastner, W.: Incremental clustering for semi-supervised anomaly detection applied on log data. In: Proceedings of the 12th International Conference on Availability, Reliability and Security, p. 31. ACM (2017)
Xu, K.S., Kliger, M., Hero III, A.O.: Adaptive evolutionary clustering. Data Min. Knowl. Discov. 28(2), 304–336 (2014)
Zhou, A., Cao, F., Qian, W., Jin, C.: Tracking clusters in evolving data streams over sliding windows. Knowl. Inf. Syst. 15(2), 181–214 (2008)
Acknowledgment
This work was partly funded by the FFG project synERGY (855457).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Landauer, M., Wurzenberger, M., Skopik, F., Settanni, G., Filzmoser, P. (2018). Time Series Analysis: Unsupervised Anomaly Detection Beyond Outlier Detection. In: Su, C., Kikuchi, H. (eds) Information Security Practice and Experience. ISPEC 2018. Lecture Notes in Computer Science(), vol 11125. Springer, Cham. https://doi.org/10.1007/978-3-319-99807-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-99807-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99806-0
Online ISBN: 978-3-319-99807-7
eBook Packages: Computer ScienceComputer Science (R0)