Abstract
Logs play an important role in system anomaly detection. However, in today’s large-scale software development and production, the cost of logging is non-negligible, and intensive logging in actual production processes will generate a large amount of redundant logs which are useless for anomaly detection. However, the current method of solving related problems is not ideal, and it is only applied when the overhead of the logging has affected the quality of service. Therefore, this paper proposes a method for optimizing logging records for this problem. Under a given budget (defined as the maximum volume of logs allowed to be output in a time interval), using an anomaly detection model based on deep learning and a two-phase filtering mechanism, the method determines whether to log according to the utility score of the log for anomaly detection to save useful logs and discard less useful logs during the system running process. The experimental results show that the proposed method alleviates the logging overhead problem without reducing the logging effectiveness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mi, H., Wang, H., Zhou, Y., et al.: Toward fine-grained, unsupervised, scalable performance diagnosis for production cloud computing systems. IEEE Trans. Parallel Distrib. Syst. 24(6), 1245–1255 (2013)
Li, H., Shang, W., Zou, Y., Hassan, A.E.: Towards just-in-time suggestions for log changes. Empirical Softw. Eng. 22(4), 1831–1865 (2016). https://doi.org/10.1007/s10664-016-9467-z
Li, H., Shang, W., Hassan, A.E.: Which log level should developers choose for a new logging statement? Empirical Softw. Eng. 22(4), 1684–1716 (2017)
Zhu, J., He, P., Qiang, F., et al.: Learning to log: helping developers make informed logging decisions. In: IEEE/ACM IEEE International Conference on Software Engineering (2015)
Xu, Z., Rodrigues, K., Yu, L., et al.: Log20: fully automated optimal placement of log printing statements under specified overhead threshold. In: The 26th Symposium (2017)
Li, H., Chen, T.H., Shang, W., et al.: Studying software logging using topic models. Empirical Softw. Eng. 7, 1–40 (2018)
Sigelman, H.B., Barroso, L., Burrows, M., et al.: Dapper, a large-scale distributed systems tracing infrastructure (2010)
Hauswirth, M., Chilimbi, T.M.: Low-overhead memory leak detection using adaptive statistical profiling. ACM Sigplan Not. 39(11), 156–164 (2004)
Arnold, M., Ryder, B.: A framework for reducing the cost of instrumented code, vol. 36 (2003)
Wei, X., Ling, H., Fox, A., et al.: Detecting large-scale system problems by mining console logs. In: ACM Sigops Symposium on Operating Systems Principles (2009)
Acknowledgement
This research is partially supported by National High-Tech Program with the Grant No. 315055101.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Y., Zheng, Q. (2021). A Logging Overhead Optimization Method Based on Anomaly Detection Model. In: Zu, Q., Tang, Y., Mladenović, V. (eds) Human Centered Computing. HCC 2020. Lecture Notes in Computer Science(), vol 12634. Springer, Cham. https://doi.org/10.1007/978-3-030-70626-5_37
Download citation
DOI: https://doi.org/10.1007/978-3-030-70626-5_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-70625-8
Online ISBN: 978-3-030-70626-5
eBook Packages: Computer ScienceComputer Science (R0)