Advertisement

An Efficient Log Parsing Algorithm Based on Heuristic Rules

  • Lin Zhang
  • Xueshuo Xie
  • Kunpeng Xie
  • Zhi Wang
  • Ye LuEmail author
  • Yujun Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11719)

Abstract

Log files usually contain very rich running information of the software system, which can be used for anomaly detection, performance modeling, and failure diagnosis, etc. In a large-scale deployment system, log records are always unstructured and can not directly use for log analysis. Log parsing, as a key prerequisite for log analysis, converts unstructured log records into structured event templates by extracting the constant portion of the raw log. Traditionally, log parsing can be achieved by manually using the regular expression, which requires many experts knowledge and has very low efficiency. Therefore, the accuracy and efficiency of log parsing are very important, especially in large-scale distributed systems. In this paper, we propose an efficient algorithm namely CLF (Clustering based on Length and First token) for extracting log event templates from raw log based on heuristic rules. The CLF algorithm works through a 3-step process: clustering unstructured logs based on heuristic rules, clustering again according to specific separation rules and finally generating event templates. Finally, we used 7 data sets to evaluate the performance of CLF and compared with three state-of-the-art log parser algorithms, where CLF ranks higher on most of the data sets and also has advantages in execution time.

Keywords

Log parsing Log analysis Clustering Algorithms 

Notes

Acknowledgment

This work is partially supported by the National Key Research and Development Program of China (2016YFC0400709), the Next Generation Internet Technology Innovation Project of CERNET (NGII20180306), the Science and Technology Commission of Tianjin Binhai New Area (BHXQKJXM-PT-ZJSHJ-2017005), the Natural Science Foundation of Tianjin (18YFYZCG00060) and Nankai University (91922299).

References

  1. 1.
    Bertero, C., Roy, M., Sauvanaud, C., Tredan, G.: Experience report: log mining using natural language processing and application to anomaly detection. In: IEEE International Symposium on Software Reliability Engineering (2017)Google Scholar
  2. 2.
    Goldstein, M., Raz, D., Segall, I.: Log-based behavioral differencing. In: IEEE International Symposium on Software Reliability Engineering, Experience report (2017)Google Scholar
  3. 3.
    Kc, K., Gu, X.: ELT: Efficient log-based troubleshooting system for cloud computing infrastructures. In: Reliable Distributed Systems (2011)Google Scholar
  4. 4.
    Ren, R., Fu, X., Zhan, J., Zhou, W.: LogMaster: Mining event correlations in logs of large scale cluster systems. In: IEEE Symposium on Reliable Distributed Systems (2012)Google Scholar
  5. 5.
    Zou, D.Q.: Uilog: improving log-based fault diagnosis by log analysis. J. Comput. Sci. Technol. 31(5), 1038–1052 (2016)CrossRefGoogle Scholar
  6. 6.
    Wong, W.E., Debroy, V., Golden, R., Xu, X., Thuraisingham, B.: Effective software fault localization using an RBF neural network. IEEE Transact. Reliab. 61(1), 149–169 (2012)CrossRefGoogle Scholar
  7. 7.
    Reidemeister, T., Jiang, M., Ward, P.A.S.: Mining unstructured log files for recurrent fault diagnosis. In: IFIP/IEEE International Symposium on Integrated Network Management (2011)Google Scholar
  8. 8.
    Nagaraj, K., Neville, J., Killian, C.: Structured comparative analysis of systems logs to diagnose performance problems. In: USENIX Conference on Networked Systems Design and Implementation (2012)Google Scholar
  9. 9.
    Fu, Q., Lou, J.G., Wang, Y., Li, J.: Execution anomaly detection in distributed systems through unstructured log analysis. In: IEEE International Conference on Data Mining (2009)Google Scholar
  10. 10.
    Lou, J.G., Fu, Q., Yang, S., Xu, Y., Li, J.: Mining invariants from console logs for system problem detection. In: Proceedings of USENIX ATC, pp. 231–244 (2010)Google Scholar
  11. 11.
    Wei, X., Ling, H., Fox, A., Patterson, D.A., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: ACM SIGOPS Symposium on Operating Systems Principles (2009)Google Scholar
  12. 12.
    Shima, K.: Length matters: clustering system log messages using length of words (2016)Google Scholar
  13. 13.
    Messaoudi, S., Panichella, A., Bianculli, D., Briand, L., Sasnauskas, R.: A search-based approach for accurate identification of log message formats. pp. 167–177 (2018)Google Scholar
  14. 14.
    Ma, H., Hellerstein, J.L.: Mining partially periodic event patterns with unknown periods. In: International Conference on Data Engineering (2000)Google Scholar
  15. 15.
    Zheng, Q., Xu, K., Lv, W., Ma, S.: Intelligent search of correlated alarms from database containing noise data. In: Network Operations and Management Symposium (2001)Google Scholar
  16. 16.
    Stearley, J.: Towards informatic analysis of syslogs. In: IEEE International Conference on Cluster Computing (2004)Google Scholar
  17. 17.
    Vaarandi, R.: A data clustering algorithm for mining patterns from event logs. In: IP Operations and Management (2003)Google Scholar
  18. 18.
    Liang, T., Tao, L., Perng, C.S.: LogSig: generating system events from raw textual logs. In: ACM International Conference on Information and Knowledge Management (2011)Google Scholar
  19. 19.
    Mizutani, M.: Incremental mining of system log format. In: IEEE International Conference on Services Computing (2013)Google Scholar
  20. 20.
    Hamooni, H., Debnath, B., Xu, J., Hui, Z., Mueen, A.: Logmine: Fast pattern recognition for log analytics. In: ACM International on Conference on Information and Knowledge Management (2016)Google Scholar
  21. 21.
    Nagappan, M., Vouk, M.A.: Abstracting log lines to log event types for mining software system logs. In: Mining Software Repositories (2010)Google Scholar
  22. 22.
    Vaarandi, R., Pihelgas, M.: LogCluster - a data clustering and pattern mining algorithm for event logs. In: International Conference on Network and Service Management (2016)Google Scholar
  23. 23.
    Jiang, Z.M., Hassan, A.E., Flora, P., Hamann, G.: Abstracting execution logs to execution events for enterprise applications (short paper). pp. 181–186, August 2008Google Scholar
  24. 24.
    Makanju, A., Zincir-Heywood, A.N., Milios, E.E.: A lightweight algorithm for message type extraction in system application logs. IEEE Transact. Knowl. Data Eng. 24(11), 1921–1936 (2012)CrossRefGoogle Scholar
  25. 25.
    He, P., Zhu, J., Zheng, Z., Lyu, M.R.: Drain: an online log parsing approach with fixed depth tree. In: IEEE International Conference on Web Services (2017)Google Scholar
  26. 26.
    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transact. Evol. Comput. 6(2), 182–197 (2002)CrossRefGoogle Scholar
  27. 27.
    Makanju, A., Zincir-Heywood, A.N., Milios, E.E.: Clustering event logs using iterative partitioning. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2009)Google Scholar
  28. 28.
    Liang, T., Tao, L.: LogTree: a framework for generating system events from raw textual logs. In: IEEE International Conference on Data Mining (2011)Google Scholar
  29. 29.
    Zhu, J., He, S., Liu, J., He, P., Lyu, M.R.: Tools and benchmarks for automated log parsing (2018)Google Scholar
  30. 30.
    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval (2010)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Lin Zhang
    • 1
  • Xueshuo Xie
    • 2
  • Kunpeng Xie
    • 2
  • Zhi Wang
    • 1
  • Ye Lu
    • 2
    Email author
  • Yujun Zhang
    • 3
  1. 1.College of Cyber ScienceNankai UniversityTianjinChina
  2. 2.College of Computer ScienceNankai UniversityTianjinChina
  3. 3.Institute of Computing TechnologyChinese Academy of SciencesBeijingChina

Personalised recommendations