Time Series Analysis: Unsupervised Anomaly Detection Beyond Outlier Detection

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11125)


Anomaly detection on log data is an important security mechanism that allows the detection of unknown attacks. Self-learning algorithms capture the behavior of a system over time and are able to identify deviations from the learned normal behavior online. The introduction of clustering techniques enabled outlier detection on log lines independent from their syntax, thereby removing the need for parsers. However, clustering methods only produce static collections of clusters. Therefore, such approaches frequently require a reformation of the clusters in dynamic environments due to changes in technical infrastructure. Moreover, clustering alone is not able to detect anomalies that do not manifest themselves as outliers but rather as log lines with spurious frequencies or incorrect periodicity. In order to overcome these deficiencies, in this paper we introduce a dynamic anomaly detection approach that generates multiple consecutive cluster maps and connects them by deploying cluster evolution techniques. For this, we design a novel clustering model that allows tracking clusters and determining their transitions. We detect anomalous system behavior by applying time-series analysis to relevant metrics computed from the evolving clusters. Finally, we evaluate our solution on an illustrative scenario and validate the achieved quality of the retrieved anomalies with respect to the runtime.


Log data Cluster evolution Anomaly detection 



This work was partly funded by the FFG project synERGY (855457).


  1. 1.
    Biggio, B., et al.: Poisoning behavioral malware clustering. In: Proceedings of the 2014 Workshop on Artificial Intelligent and Security Workshop, pp. 27–36. ACM (2014)Google Scholar
  2. 2.
    Chakrabarti, D., Kumar, R., Tomkins, A.: Evolutionary clustering. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 554–560. ACM (2006)Google Scholar
  3. 3.
    Chan, J., Bailey, J., Leckie, C.: Discovering correlated spatio-temporal changes in evolving graphs. Knowl. Inf. Syst. 16(1), 53–96 (2008)CrossRefGoogle Scholar
  4. 4.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 15 (2009)CrossRefGoogle Scholar
  5. 5.
    Chi, Y., Song, X., Zhou, D., Hino, K., Tseng, B.L.: On evolutionary spectral clustering. ACM Trans. Knowl. Discov. Data (TKDD) 3(4), 17 p. (2009)Google Scholar
  6. 6.
    Cryer, J., Chan, K.: Time Series Analysis: With Applications in R. Springer Texts in Statistics. Springer, Heidelberg (2008). Scholar
  7. 7.
    Greene, D., Doyle, D., Cunningham, P.: Tracking the evolution of communities in dynamic social networks. In: Advances in Social Networks Analysis and Mining (ASONAM), pp. 176–183. IEEE (2010)Google Scholar
  8. 8.
    He, S., Zhu, J., He, P., Lyu, M.R.: Experience report: system log analysis for anomaly detection. In: 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pp. 207–218. IEEE (2016)Google Scholar
  9. 9.
    Jensen, C.S., Lin, D., Ooi, B.C.: Continuous clustering of moving objects. IEEE Trans. Knowl. Data Eng. 19(9), 1161–1174 (2007)CrossRefGoogle Scholar
  10. 10.
    Killick, R., Fearnhead, P., Eckley, I.A.: Optimal detection of changepoints with a linear computational cost. J. Am. Stat. Assoc. 107(500), 1590–1598 (2012)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Lughofer, E., Sayed-Mouchaweh, M.: Autonomous data stream clustering implementing split-and-merge concepts-towards a plug-and-play approach. Inf. Sci. 304, 54–79 (2015)CrossRefGoogle Scholar
  12. 12.
    Pincombe, B.: Anomaly detection in time series of graphs using arma processes. Asor Bull. 24(4), 2 (2005)Google Scholar
  13. 13.
    Scarfone, K., Mell, P.: Guide to intrusion detection and prevention systems (IDPS). NIST Special Publication 800-94 (2007)Google Scholar
  14. 14.
    Skopik, F., Settanni, G., Fiedler, R., Friedberg, I.: Semi-synthetic data set generation for security software evaluation. In: 2014 Twelfth Annual International Conference on Privacy, Security and Trust (PST), pp. 156–163. IEEE (2014)Google Scholar
  15. 15.
    Spiliopoulou, M., Ntoutsi, I., Theodoridis, Y., Schult, R.: MONIC: modeling and monitoring cluster transitions. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 706–711. ACM (2006)Google Scholar
  16. 16.
    Toyoda, M., Kitsuregawa, M.: Extracting evolution of web communities from a series of web archives. In: Proceedings of the Fourteenth ACM Conference on Hypertext and Hypermedia, pp. 28–37. ACM (2003)Google Scholar
  17. 17.
    Wurzenberger, M., Skopik, F., Landauer, M., Greitbauer, P., Fiedler, R., Kastner, W.: Incremental clustering for semi-supervised anomaly detection applied on log data. In: Proceedings of the 12th International Conference on Availability, Reliability and Security, p. 31. ACM (2017)Google Scholar
  18. 18.
    Xu, K.S., Kliger, M., Hero III, A.O.: Adaptive evolutionary clustering. Data Min. Knowl. Discov. 28(2), 304–336 (2014)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Zhou, A., Cao, F., Qian, W., Jin, C.: Tracking clusters in evolving data streams over sliding windows. Knowl. Inf. Syst. 15(2), 181–214 (2008)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Austrian Institute of TechnologyViennaAustria
  2. 2.Vienna University of TechnologyViennaAustria

Personalised recommendations