Abstract
As the number of logs generated by each node in the national high-performance computing environment continues to increase, manual analysis of abnormal logs has become slow and inaccurate. This makes it difficult to meet the requirements of daily system analysis. In response to this problem, this paper proposes a method for defining an abnormal log business model. Through the analysis of the exception logs of the multi-node system, this paper finds that the exception logs have a certain regularity and repeatability. Therefore, the abnormal logs can be detected by analyzing the log traffic pattern. This method uses machine learning algorithm to analyze log data in multi-node system, extract normal log traffic patterns, and use these patterns to detect abnormal behavior. Experimental results show that the proposed method can effectively detect abnormal log traffic patterns in multi-node systems, with high accuracy and robustness, and can detect and locate system faults in time, improving the reliability and stability of the system. This paper provides a new solution for log traffic pattern detection of multi-node system, which has a certain application prospect in the field of computer system monitoring, and can provide a certain reference for computer system monitoring and maintenance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data will be available upon request.
References
Augusto, A., Conforti, R., Dumas, M., et al.: Automated discovery of process models from event logs: review and benchmark. IEEE Trans. Knowl. Data Eng. 31(4), 686–705 (2018)
Bao, W., Lianju, N., Yue, K.: Integration of unsupervised and supervised machine learning algorithms for credit risk assessment. Expert Syst. Appl. 128, 301–315 (2019)
Bhanage, D.A., Pawar, A.V., Kotecha, K.: It infrastructure anomaly detection and failure handling: a systematic literature review focusing on datasets, log preprocessing, machine & deep learning approaches and automated tool. IEEE Access 9, 156392–156421 (2021)
Deokar, A.V., Tao, J.: Semantics-based event log aggregation for process mining and analytics. Inf. Syst. Front. 17, 1209–1226 (2015)
Homayoun, S., Dehghantanha, A., Ahmadzadeh, M., Hashemi, S., Khayami, R.: Know abnormal, find evil: frequent pattern mining for ransomware threat hunting and intelligence. IEEE Trans. Emerg. Top. Comput. 8(2), 341–351 (2017)
Janani, R., Vijayarani, S.: Text document clustering using spectral clustering algorithm with particle swarm optimization. Expert Syst. Appl. 134, 192–200 (2019)
Jin, X., Sun, J., Zhang, R., Zhang, Y., Zhang, C.: Specguard: Spectrum misuse detection in dynamic spectrum access systems. IEEE Trans. Mob. Comput. 17(12), 2925–2938 (2018)
Karimi-Maleh, H., Karimi, F., Alizadeh, M., Sanati, A.L.: Electrochemical sensors, a bright future in the fabrication of portable kits in analytical systems. Chem. Rec. 20(7), 682–692 (2020)
Khraisat, A., Gondal, I., Vamplew, P., Kamruzzaman, J.: Survey of intrusion detection systems: techniques, datasets and challenges. Cybersecurity 2(1), 1–22 (2019)
Landauer, M., Skopik, F., Wurzenberger, M., Rauber, A.: System log clustering approaches for cyber security applications: a survey. Comput. Secur. 92, 101739 (2020)
Marin-Castro, H.M., Tello-Leal, E.: Event log preprocessing for process mining: a review. Appl. Sci. 11(22), 10556 (2021)
Pei, J., Zhong, K., Jan, M.A., Li, J.: Personalized federated learning framework for network traffic anomaly detection. Comput. Netw. 209, 108906 (2022)
Salehi, H., Burgueño, R.: Emerging artificial intelligence methods in structural engineering. Eng. Struct. 171, 170–189 (2018)
Sanusi, I.T., Oyelere, S.S., Omidiora, J.O.: Exploring teachers’ preconceptions of teaching machine learning in high school: a preliminary insight from Africa. Comput. Educ. Open 3, 100072 (2022)
Saranya, T., Sridevi, S., Deisy, C., Chung, T.D., Khan, M.A.: Performance analysis of machine learning algorithms in intrusion detection system: a review. Proced. Comput. Sci. 171, 1251–1260 (2020)
Funding
This paper was supported by (1) Characteristic Innovation Project of Colleges and Universities of Department of Education of Guangdong Province: Research on the Technology of Urban Intelligent Security Guarantee Pedestrian Flow Monitoring Platform Based on Big Data (No. 2022KTSCX263); (2) Support Project for Scientific Research Workers in Shaoguan of Guandong Povince (No. 230328228030756); (3) Project Fund of Guangdong Songshan Polytechnic (No. 2021KJYB004, 2022JYJG06,). (4)Education and Teaching Reform Project of Guangdong Higher Vocational Public Security Judicial and Public Management Education and Guidance Committee (No.2022YL06) (5) Supported for Project of importent field for general universities in Guangdong Province (No.2021ZDZX4109).
Author information
Authors and Affiliations
Contributions
The first version was written by JC, BP and XZ has done the simulations. All authors have contributed to the paper’s analysis, discussion, writing, and revision.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no competing interests.
Ethical approval
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cao, J., Pan, B. & Zou, X. Flow monitoring system and abnormal log traffic mode detection based on artificial intelligence. Opt Quant Electron 56, 112 (2024). https://doi.org/10.1007/s11082-023-05690-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11082-023-05690-z