Abstract
A key source of information describing a system’s current state is log data. However, accessing this information for further analysis is often complicated. Usually, log data is available in form of unstructured text lines and there exists no common standard for the appearance of logs. Hence, log parsers are required to pre-process log lines and structure their information for further analysis. State of the art log parsers still apply pre-defined lists of regular expressions, which are linearly processed and thus hinder online log analysis. Furthermore, defining log parsers manually is a cumbersome and time consuming task. Therefore, in this chapter we propose AECID-PG, a novel log parser generator. AECID-PG implements a density-based approach to automatically generate a tree-like parser, which reduces the complexity of log parsing significantly.
Parts of this chapter have been published in [120].
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A more exhaustive list of model elements can be found in the AMiner (which is an agent that can apply the parser) documentation at: https://github.com/ait-aecid/logdata-anomaly-miner/blob/V2.2.3/source/root/usr/share/doc/logdata-anomaly-miner/aminer/ParsingModel.txt.
- 2.
- 3.
- 4.
Note that the implementation of the aecid-parsergenerator demonstrated in this try-it-out includes new features that go beyond the theoretical discussions of the previous sections. In particular, the third rule stated in Sect. 7.3.3 is updated to support combinations of static and variable nodes, and a branch similarity metric is used to merge nodes followed by similar sub-trees. Further details can be found in the repository of the aecid-parsergenerator.
References
R Gerhards. The syslog protocol: Rfc 5424. IETF Trust: Reston, VA, USA, 2009.
Risto Vaarandi. A data clustering algorithm for mining patterns from event logs. In Proceedings of the 3rd Workshop on IP Operations & Management (IPOM 2003), pages 119–126. IEEE, 2003.
Markus Wurzenberger, Max Landauer, Florian Skopik, and Wolfgang Kastner. Aecid-pg: A tree-based log parser generator to enable log analysis. In 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), pages 7–12. IEEE, 2019.
Markus Wurzenberger, Florian Skopik, Giuseppe Settanni, and Roman Fiedler. Aecid: A self-learning anomaly detection approach based on light-weight log parser models. In ICISSP, pages 386–397, 2018.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Skopik, F., Wurzenberger, M., Landauer, M. (2021). A Concept for a Tree-Based Log Parser Generator. In: Smart Log Data Analytics. Springer, Cham. https://doi.org/10.1007/978-3-030-74450-2_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-74450-2_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-74449-6
Online ISBN: 978-3-030-74450-2
eBook Packages: Computer ScienceComputer Science (R0)