Skip to main content
SpringerLink
Log in

On the effectiveness of log representation for log-based anomaly detection

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Logs are an essential source of information for people to understand the running status of a software system. Due to the evolving modern software architecture and maintenance methods, more research efforts have been devoted to automated log analysis. In particular, machine learning (ML) has been widely used in log analysis tasks. In ML-based log analysis tasks, converting textual log data into numerical feature vectors is a critical and indispensable step. However, the impact of using different log representation techniques on the performance of the downstream models is not clear, which limits researchers and practitioners’ opportunities of choosing the optimal log representation techniques in their automated log analysis workflows. Therefore, this work investigates and compares the commonly adopted log representation techniques from previous log analysis research. Particularly, we select six log representation techniques and evaluate them with seven ML models and four public log datasets (i.e., HDFS, BGL, Spirit and Thunderbird) in the context of log-based anomaly detection.We also examine the impacts of the log parsing process and the different feature aggregation approaches when they are employed with log representation techniques. From the experiments, we provide some heuristic guidelines for future researchers and developers to follow when designing an automated log analysis workflow. We believe our comprehensive comparison of log representation techniques can help researchers and practitioners better understand the characteristics of different log representation techniques and provide them with guidance for selecting the most suitable ones for their ML-based log analysis workflow.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Scripts and data files used in our research are available online and can be found in our replication package: https://github.com/mooselab/suppmaterial-LogRepForAnomalyDetection.

  2. https://code.google.com/archive/p/word2vec/

References

  • Chen M, Zheng AX, Lloyd J, Jordan MI, Brewer E (2004) Failure diagnosis using decision trees. In International Conference on Autonomic Computing, 2004. Proceedings., pages 36–43. IEEE

  • Chen Z, Liu J, Gu W, Su Y, Lyu MR (2021) Experience report: Deep learning-based system log analysis for anomaly detection. arXiv preprint arXiv:2107.05908

  • Chow M, Meisner D, Flinn J, Peek D, Wenisch TF (2014) The mystery machine: End-to-end performance analysis of large-scale internet services. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 217–231

  • Dai H, Li H, Chen CS, Shang W, Chen T-H (2020) Logram: Efficient log parsing using n-gram dictionaries. IEEE Transactions on Software Engineering 48(3):879–892. https://doi.org/10.1109/TSE.2020.3007554

  • Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert:Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  • Du M, Li F, Zheng G, Srikumar V (2017) Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 1285–1298

  • El-Sayed N, Zhu H, Schroeder B (2017) Learning from failure across multiple clusters: A trace-driven approach to understanding, predicting, and mitigating job terminations. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pages 1333–1344. IEEE

  • Fu Q, Lou J-G, Wang Y, Li J (2009) Execution anomaly detection in distributed systems through unstructured log analysis. In 2009 ninth IEEE international conference on data mining, pages 149–158. IEEE

  • Fu Q, Lou J-G, Lin Q, Ding R, Zhang D, Xie T (2013) Contextual analysis of program logs for understanding system behaviors. In 2013 10th Working Conference on Mining Software Repositories (MSR), pages 397– 400. IEEE.

  • Grave E, Bojanowski P, Gupta P, Joulin A, Mikolov T (2018) Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)

  • Hansen SE, Atkins ET (1993) Automated system monitoring and notification with swatch. In LISA, volume 93, pages 145–152. Monterey, CA

  • He S, He P, Chen Z, Yang T, Su Y, Lyu MR (2021) A survey on automated log analysis for reliability engineering. ACM Comput Sur (CSUR) 54(6):1–37

    Google Scholar 

  • He P, Zhu J, He S, Li J, Lyu MR (2016a) An evaluation study on log parsing and its use in log mining. In 2016a 46th annual IEEE/IFIP international conference on dependable systems and networks (DSN), pages 654–661. IEEE

  • He S, Zhu J, He P, Lyu MR (2016b) Experience report: System log analysis for anomaly detection. In 2016b IEEE 27th international symposium on software reliability engineering (ISSRE), pages 207–218. IEEE

  • He P, Zhu J, Zheng Z, Lyu MR (2017) Drain: An online log parsing approach with fixed depth tree. In 2017 IEEE international conference on web services (ICWS), pages 33–40. IEEE

  • He S, Zhu J, He P, Lyu MR (2020) Loghub: a large collection of system log datasets towards automated log analytics.arXiv preprint arXiv:2008.06448

  • Jarry R, Kobayashi S, Fukuda K (2021) A quantitative causal analysis for network log data. In 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), pages 1437–1442. IEEE

  • Katkar DGS, Kasliwal AD (2014) Use of log data for predictive analytics through data mining. Current Trends in Technology and Science. Volume: 3, Issue: 3 (Apr-May 2014)

  • Khan ZA, Shin D, Bianculli D, Briand L (2022) Guidelines for assessing the accuracy of log message template identification techniques. In Proceedings of the 44th International Conference on Software Engineering, pages 1095–1106

  • Le V-H, Zhang H (2021) Log-based anomaly detection without log parsing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 492–504. IEEE.

  • Le V-H, Zhang H (2022) Log-based anomaly detection with deep learning: how far are we? In 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), pages 1356–1367. IEEE

  • Le V-H, Zhang H (2023) Log parsing with prompt-based few-shot learning. arXiv preprint arXiv:2302.07435.

  • Li X, Chen P, Jing L, He Z, Yu G (2020) Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pages 92–103. IEEE.

  • Liang Y, Zhang Y, Xiong H, Sahoo R (2007) Failure prediction in ibm bluegene/l event logs. In Seventh IEEE International Conference on Data Mining (ICDM 2007), pages 583–588. IEEE.

  • Liao L, Chen J, Li H, Zeng Y, Shang W, Guo J, Sporea C, Toma A, Sajedi S (2020) Using black-box performance models to detect performance regressions under varying workloads: an empirical study. Empir Softw Eng 25(5):4130–4160

    Article  Google Scholar 

  • Liu FT, Ting KM, Zhou Z-H (2012) Isolation-based anomaly detection. ACM Trans Knowl Discov Data (TKDD) 6(1):1–39

    Article  Google Scholar 

  • Liu Y, Zhang X, He S, Zhang H, Li L, Kang Y, Xu Y, Ma M, Lin Q, Dang Y et al (2022) Uniparser: A unified log parser for heterogeneous log data. Proc ACM Web Conf 2022:1893–1901

    Google Scholar 

  • Lou J-G, Fu Q, Yang S, Xu Y, Li J (2010) Mining invariants from console logs for system problem detection. In 2010 USENIX Annual Technical Conference (USENIX ATC 10)

  • Lu S, Wei X, Li Y, Wang L (2018) Detecting anomaly in big data system logs using convolutional neural network. In 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/- DataCom/CyberSciTech), pages 151–158. IEEE

  • Lyu Y, Li H, Sayagh M, Jiang ZM, Hassan AE (2021) An empirical study of the impact of data splitting decisions on the performance of aiops solutions. ACM Trans Softw Eng Methodol (TOSEM) 30(4):1–38

    Article  Google Scholar 

  • Meng W, Liu Y, Zhu Y, Zhang S, Pei D, Liu Y, Chen Y, Zhang R, Tao S, Sun P et al (2019) Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. In IJCAI 19:4739–4745

    Google Scholar 

  • Meng W, Liu Y, Zhang S, Zaiter F, Zhang Y, Huang Y, Yu Z, Zhang Y, Song L, Zhang M et al (2021) Logclass: Anomalous log identification and classification with partial labels. IEEE Trans Netw Serv Manage 18(2):1870–1884

    Article  Google Scholar 

  • Nagaraj K, Killian C, Neville J (2012) Structured comparative analysis of systems logs to diagnose performance problems. In 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12), pages 353–366

  • Nedelkoski S, Bogatinovski J, Acker A, Cardoso J, Kao O (2020) Self-attentive classification-based anomaly detection in unstructured logs. In 2020 IEEE International Conference on Data Mining (ICDM), pages 1196–1201. IEEE

  • Nguyen KA,Walde SSi, Vu NT (2016) Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction. arXiv preprint arXiv:1605.07766.

  • Oliner A, Ganapathi A, Xu W (2012) Advances and challenges in log analysis. Commun ACM 55(2):55–61

    Article  Google Scholar 

  • Oliner A, Stearley J (2007) What supercomputers say: A study of five system logs. In 37th annual IEEE/IFIP international conference on dependable systems and networks (DSN’07), pages 575–584. IEEE

  • Prewett JE (2003) Analyzing cluster log files using logsurfer. In Proceedings of the 4th Annual Conference on Linux Clusters. Citeseer

  • Rouillard JP (2004) Real-time log file analysis using the simple event correlator (sec). In LISA 4:133–150

    Google Scholar 

  • Rusticus SA, Lovato CY (2014) Impact of sample size and variability on the power and type i error rates of equivalence tests: A simulation study. Pract Assess Res Eval 19(1):11

    Google Scholar 

  • Salton G, Buckley C (1988) Term-weighting approaches in automatic text retrieval. Inf Process Manage 24(5):513–523

    Article  Google Scholar 

  • Schroeder B, Gibson GA (2007) Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? In 5th USENIX Conference on File and Storage Technologies (FAST 07), San Jose, CA. USENIX Association

  • Shang W, Jiang ZM, Adams B, Hassan AE, Godfrey MW, Nasser M, Flora P (2014) An exploratory study of the evolution of communicated information about the execution of large software systems. J Softw: Evol Process 26(1):3–26

    Google Scholar 

  • Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2017) An empirical comparison of model validation techniques for defect prediction models. IEEE Transactions on Software Engineering 43(1):1–18.  https://doi.org/10.1109/TSE.2016.2584050

  • Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2018) The Impact of Automated Parameter Optimization on Defect Prediction Models. IEEE Transactions on Software Engineering 45(7):683–711. https://doi.org/10.1109/TSE.2018.2794977

  • Turc I, Chang M-W, Lee K, Toutanova K (2019) Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962v2

  • Van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11)

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

  • Wan Y, Liu Y, Wang D, Wen Y (2021) Glad-paw: Graph-based log anomaly detection by position aware weighted graph attention network. In Advances in Knowledge Discovery and Data Mining: 25th Pacific-Asia Conference, PAKDD 2021, Virtual Event, May 11–14, 2021, Proceedings, Part I , pages 66–77. Springer

  • Wang M, Xu L, Guo L (2018) Anomaly detection of system logs based on natural language processing and deep learning. In 2018 4th International Conference on Frontiers of Signal Processing (ICFSP), pages 140–144. IEEE

  • Xie Y, Zhang H, Babar MA (2022) Loggd: Detecting anomalies from system logs by graph neural networks. arXiv preprint arXiv:2209.07869

  • Xu W, Huang L, Fox A, Patterson D, Jordan MI (2009) Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pages 117– 132

  • Yuan D, Mai H, Xiong W, Tan L, Zhou Y, Pasupathy S (2010) Sherlog: error diagnosis by connecting clues from run-time logs. In Proceedings of the fifteenth International Conference on Architectural support for programming languages and operating systems, pages 143–154

  • Yuan D, Park S, Zhou Y (2012) Characterizing logging practices in open-source software. In 2012 34th International Conference on Software Engineering (ICSE), pages 102–112. IEEE

  • Zhang X, Xu Y, Lin Q, Qiao B, Zhang H, Dang Y, Xie C, Yang X, Cheng Q, Li Z, et al. (2019) Robust log-based anomaly detection on unstable log data. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 807–817

  • Zhu J, He S, Liu J, He P, Xie Q, Zheng Z, Lyu MR (2019) Tools and benchmarks for automated log parsing. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 121–130. IEEE

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering and Software Engineering, Polytechnique Montreal, Montreal, QC, Canada

    Xingfang Wu, Heng Li & Foutse Khomh

Authors
  1. Xingfang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Foutse Khomh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xingfang Wu.

Ethics declarations

Conflicts of interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Communicated by: Mika Mäntylä

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, X., Li, H. & Khomh, F. On the effectiveness of log representation for log-based anomaly detection. Empir Software Eng 28, 137 (2023). https://doi.org/10.1007/s10664-023-10364-1

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-023-10364-1

Keywords

Search

Navigation