Abstract
With the continuous increase in data velocity and volume nowadays, preserving system and data security is particularly affected. In order to handle the huge amount of data and to discover security incidents in real-time, analyses of log data streams are required. However, most of the log anomaly detection techniques fall short in considering continuous data processing. Thus, this paper aligns an anomaly detection technique for data stream processing. It thereby provides a conceptual basis for future adaption of other techniques and further delivers proof of concept by prototype implementation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
A project funded by the German Ministry of Education and Research that provides real-world log streams of various systems within an enterprise infrastructure.
- 3.
- 4.
- 5.
References
Agarwal, D.: Detecting anomalies in cross-classified streams: a Bayesian approach. Knowl. Inf. Syst. 11(1), 29–44 (2006)
Andrade, H.C.M., Gedik, B., Turaga, D.S.: Fundamentals of Stream Processing: Application Design, Systems, and Analytics. Cambridge University Press, Cambridge (2014)
Angiulli, F., Fassetti, F.: Detecting distance-based outliers in streams of data. In: Silva, M.J., et al. (eds.) Proceedings of the 2007 ACM Conference on Information and Knowledge Management, p. 811. ACM, New York (2007)
Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: Abiteboul, S. (ed.) Proceedings of the Twenty-first ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 1–30. ACM, New York (2002)
Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection. ACM Comput. Surv. 41(3), 1–58 (2009)
Chuvakin, A., Schmidt, K., Phillips, C.: Logging and Log Management: The Authoritative Guide to Dealing with Syslog, Audit Logs, Events, Alerts and other IT ’Noise’. Elsevier Science, Burlington (2012)
Cugola, G., Margara, A.: Processing flows of information. ACM Comput. Surv. 44(3), 1–62 (2012)
Golab, L., Özsu, M.T.: Issues in data stream management. ACM SIGMOD Rec. 32(2), 5–14 (2003)
Hébrail, G.: Data stream management and mining. In: Fogelmann-Soulié, F., Perrotta, D., Piskorski, J., Steinberger, R. (eds.) Mining Massive Data Sets for Security, pp. 89–102. IOS Press (2008)
Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS Q. 28(1), 75–105 (2004)
Hussain, A.R., Hameed, M.A., Fatima, S.: A proposal: high-throughput robust architecture for log analysis and data stream mining. In: Saini, H.S., Sayal, R., Rawat, S.S. (eds.) Innovations in Computer Science and Engineering. AISC, vol. 413, pp. 305–314. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-0419-3_36
Kent, K., Souppaya, M.P.: Guide to computer security log management. Technical report, National Institute of Standards and Technology, Gaithersburg, MD. https://www.nist.gov/publications/guide-computer-security-log-management
Korolov, M.: Log management is leading use case for big data (2015). https://www.csoonline.com/article/2935362/data-protection/log-management-is-leading-use-case-for-big-data.html
National Institute of Standards and Technology: Big data interoperability framework: Volume 1, defintions. https://bigdatawg.nist.gov/_uploadfiles/NIST.SP.1500-1.pdf
Neely, L.: 2017 threat landscape survey: users on the front line. Technical report, SANS Institue. https://www.sans.org/reading-room/whitepapers/threats/2017-threat-landscape-survey-users-front-line-37910
Nunamaker, J.F., Chen, M.: Systems development in information systems research. In: Twenty-Third Annual Hawaii International Conference on System Sciences, pp. 631–640. IEEE Computer Society Press (1990)
Olbrich, S.: Warehousing and analyzing streaming data quality information. In: AMCIS 2010 Proceedings (2010)
Sadik, S., Gruenwald, L.: Research issues in outlier detection for data streams. ACM SIGKDD Explor. Newslett. 15(1), 33–40 (2014)
Winding, R., Wright, T., Chapple, M.: System anomaly detection: mining firewall logs. In: 2006 SecureComm and workshops, pp. 1–5. IEEE, Piscataway, NJ (2006)
Acknowledgements
Part of this research was supported by the Federal Ministry of Education and Research, Germany, as part of the BMBF DINGfest project (https://dingfest.ur.de).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Dietz, M., Pernul, G. (2018). Big Log Data Stream Processing: Adapting an Anomaly Detection Technique. In: Hartmann, S., Ma, H., Hameurlain, A., Pernul, G., Wagner, R. (eds) Database and Expert Systems Applications. DEXA 2018. Lecture Notes in Computer Science(), vol 11030. Springer, Cham. https://doi.org/10.1007/978-3-319-98812-2_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-98812-2_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-98811-5
Online ISBN: 978-3-319-98812-2
eBook Packages: Computer ScienceComputer Science (R0)