Abstract
Log analysis is an important technique that engineers use for troubleshooting faults of large-scale service-oriented systems. In this study, we propose a novel semi-supervised log-based anomaly detection approach, LogDP, which utilizes the dependency relationships among log events and proximity among log sequences to detect the anomalies in massive unlabeled log data. LogDP divides log events into dependent and independent events, then learns the normal patterns of dependent events based on the dependencies among events and the normal patterns of independent events based on the deviation of values from a historic mean. Events violating any normal pattern are identified as anomalies. By combining dependency and proximity, LogDP is able to achieve high detection accuracy. Extensive experiments have been conducted on real-world datasets, and the results show that LogDP outperforms six state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Lou, J.G., Fu, Q., Yang, S., Xu, Y., Li, J.: Mining invariants from console logs for system problem detection. In: USENIX Annual Technical Conference (2010)
He, S., Zhu, J., He, P., Lyu, M.: Experience report: system log analysis for anomaly detection. In: ISSRE, pp. 207–218. IEEE (2016)
Zhang, B., Zhang, H., Moscato, P., Zhang, A.: Anomaly detection via mining numerical workflow relations from logs. In: SRDS. IEEE (2020)
Du, M., Li, F., Zheng, G., Srikumar, V.: Deeplog: anomaly detection and diagnosis from system logs through deep learning. In: CCS (2017)
Meng, W., et al.: Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI 2019, pp. 4739–4745 (2019)
Le, V.H., Zhang, H.: Log-based anomaly detection without log parsing. In: Proceedings of the 2021 IEEE/ACM Automated Software Engineering Conference, ASE (2021)
Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: SOSP, pp. 117–132 (2009)
Schölkopf, B., Platt, J.C., Shawe-Taylor, J., Smola, A.J., Williamson, R.C.: Estimating the support of a high-dimensional distribution. Neural Comput. 13(7), 1443–1471 (2001)
Lin, Q., Zhang, H., Lou, J., Zhang, Yu., Chen, X.: Log clustering based problem identification for online service systems. In: ICSE-C. IEEE (2016)
He, P., Zhu, J., Zheng, Z., Lyu, M.: Drain: an online log parsing approach with fixed depth tree. In: ICWS. IEEE (2017)
Du, M., Li, F.: Spell: streaming parsing of system event logs. In: IEEE ICDM, pp. 859–864. IEEE (2016)
Dai, H., Li, H., Chen, C., Shang, W., Chen, T.: Logram: efficient log parsing using n-gram dictionaries. IEEE Trans. Softw. Eng. (2020)
Lu, S., Liu, L., Li, J., Le, T.D., Liu, J.: Lopad: a local prediction approach to anomaly detection. In: Advances in Knowledge Discovery and Data Mining (2020)
Pearl, J.: Causality: Models, Reasoning and Inference. Springer, Heidelberg (2000)
He, S., Zhu, J., He, P., Lyu, M.R.: Loghub: a large collection of system log datasets towards automated log analytics. arXiv e-prints (2020)
Acknowledgments
This research was supported by an Australian Government Research Training Program (RTP) Scholarship, and by the Australian Research Council’s Discovery Projects funding scheme (project DP200102940). The work was also supported with super-computing resources provided by the Phoenix High Powered Computing (HPC) service at the University of Adelaide.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Xie, Y., Zhang, H., Zhang, B., Babar, M.A., Lu, S. (2021). LogDP: Combining Dependency and Proximity for Log-Based Anomaly Detection. In: Hacid, H., Kao, O., Mecella, M., Moha, N., Paik, Hy. (eds) Service-Oriented Computing. ICSOC 2021. Lecture Notes in Computer Science(), vol 13121. Springer, Cham. https://doi.org/10.1007/978-3-030-91431-8_47
Download citation
DOI: https://doi.org/10.1007/978-3-030-91431-8_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-91430-1
Online ISBN: 978-3-030-91431-8
eBook Packages: Computer ScienceComputer Science (R0)