Big Log Data Stream Processing: Adapting an Anomaly Detection Technique

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11030)


With the continuous increase in data velocity and volume nowadays, preserving system and data security is particularly affected. In order to handle the huge amount of data and to discover security incidents in real-time, analyses of log data streams are required. However, most of the log anomaly detection techniques fall short in considering continuous data processing. Thus, this paper aligns an anomaly detection technique for data stream processing. It thereby provides a conceptual basis for future adaption of other techniques and further delivers proof of concept by prototype implementation.


Data stream Anomaly detection Log analysis Real-time analysis 



Part of this research was supported by the Federal Ministry of Education and Research, Germany, as part of the BMBF DINGfest project (


  1. 1.
    Agarwal, D.: Detecting anomalies in cross-classified streams: a Bayesian approach. Knowl. Inf. Syst. 11(1), 29–44 (2006)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Andrade, H.C.M., Gedik, B., Turaga, D.S.: Fundamentals of Stream Processing: Application Design, Systems, and Analytics. Cambridge University Press, Cambridge (2014)CrossRefGoogle Scholar
  3. 3.
    Angiulli, F., Fassetti, F.: Detecting distance-based outliers in streams of data. In: Silva, M.J., et al. (eds.) Proceedings of the 2007 ACM Conference on Information and Knowledge Management, p. 811. ACM, New York (2007)Google Scholar
  4. 4.
    Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: Abiteboul, S. (ed.) Proceedings of the Twenty-first ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 1–30. ACM, New York (2002)Google Scholar
  5. 5.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection. ACM Comput. Surv. 41(3), 1–58 (2009)CrossRefGoogle Scholar
  6. 6.
    Chuvakin, A., Schmidt, K., Phillips, C.: Logging and Log Management: The Authoritative Guide to Dealing with Syslog, Audit Logs, Events, Alerts and other IT ’Noise’. Elsevier Science, Burlington (2012)Google Scholar
  7. 7.
    Cugola, G., Margara, A.: Processing flows of information. ACM Comput. Surv. 44(3), 1–62 (2012)CrossRefGoogle Scholar
  8. 8.
    Golab, L., Özsu, M.T.: Issues in data stream management. ACM SIGMOD Rec. 32(2), 5–14 (2003)CrossRefGoogle Scholar
  9. 9.
    Hébrail, G.: Data stream management and mining. In: Fogelmann-Soulié, F., Perrotta, D., Piskorski, J., Steinberger, R. (eds.) Mining Massive Data Sets for Security, pp. 89–102. IOS Press (2008)Google Scholar
  10. 10.
    Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS Q. 28(1), 75–105 (2004)CrossRefGoogle Scholar
  11. 11.
    Hussain, A.R., Hameed, M.A., Fatima, S.: A proposal: high-throughput robust architecture for log analysis and data stream mining. In: Saini, H.S., Sayal, R., Rawat, S.S. (eds.) Innovations in Computer Science and Engineering. AISC, vol. 413, pp. 305–314. Springer, Singapore (2016). Scholar
  12. 12.
    Kent, K., Souppaya, M.P.: Guide to computer security log management. Technical report, National Institute of Standards and Technology, Gaithersburg, MD.
  13. 13.
  14. 14.
    National Institute of Standards and Technology: Big data interoperability framework: Volume 1, defintions.
  15. 15.
    Neely, L.: 2017 threat landscape survey: users on the front line. Technical report, SANS Institue.
  16. 16.
    Nunamaker, J.F., Chen, M.: Systems development in information systems research. In: Twenty-Third Annual Hawaii International Conference on System Sciences, pp. 631–640. IEEE Computer Society Press (1990)Google Scholar
  17. 17.
    Olbrich, S.: Warehousing and analyzing streaming data quality information. In: AMCIS 2010 Proceedings (2010)Google Scholar
  18. 18.
    Sadik, S., Gruenwald, L.: Research issues in outlier detection for data streams. ACM SIGKDD Explor. Newslett. 15(1), 33–40 (2014)CrossRefGoogle Scholar
  19. 19.
    Winding, R., Wright, T., Chapple, M.: System anomaly detection: mining firewall logs. In: 2006 SecureComm and workshops, pp. 1–5. IEEE, Piscataway, NJ (2006)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Universität Regensburg, Lehrstuhl für Wirtschaftinformatik IRegensburgGermany

Personalised recommendations