Abstract
Log management plays an essential role in identifying problems and troubleshoot problems in a distributed system. However, when we conducted log analysis on big data cluster, Kubernetes cluster and Ai capability cluster, we found it was difficult to find a Distributed cloud monitoring platform that met our requirements. So, we propose a Distributed cloud monitoring platform based on log insight, which can be used to achieve unified log insight of big data clusters, K8s clusters, and Ai capability clusters. At the same time, through this system, Developers can intuitively monitor and analyze the business system data and cluster operation monitoring data. Once there is a problem in the log, it will immediately alert, locate, display, and track the message. This system is helpful to improve the readability of log information to administrators, In the process of data collection, Filebeat and Metricbeat will be combined to collect data, therefore, the system can not only collect ordinary log data but also support to collect the indicator data of each famous mature system (Such as operating system, Memcached, Mysql, Docker, Kafka, etc.). Besides, the system will monitor and manage the status of cluster nodes through BeatWatcher. Finally, we develop the system and verify its feasibility and performance by simulation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kononenko, O., et al.: Mining modern repositories with elasticsearch. In: MSR (2014)
Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Comput. Surv. 38, 6 (2006)
Prakash, T.R., et al.: Geo-identification of web users through logs using ELK stack. In: 2016 6th International Conference - Cloud System and Big Data Engineering (Confluence), pp. 606–610 (2016)
Montesi, F., Weber, J.: Circuit breakers, discovery, and API gateways in microservices. CoRR abs/1609.05830 (2016)
Arpitha, P., Kumar, P.V.: Big data computing and clouds: trends and future directions (2018)
Carbone, P., et al.: Apache Flink™: stream and batch processing in a single engine. IEEE Data Eng. Bull. 38, 28–38 (2015)
Carbone, P., et al.: State management in apache Flink®: consistent stateful distributed stream processing. PVLDB 10, 1718–1729 (2017)
Tovarnák, D., Pitner, T.: Normalization of unstructured log data into streams of structured event objects. In: 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), pp. 671–676 (2019)
Tang, J., et al.: Visualizing large-scale and high-dimensional data. In: WWW (2016)
Dumais, S., Jeffries, R., Russell, D.M., Tang, D., Teevan, J.: Understanding user behavior through log data and analysis. In: Olson, J.S., Kellogg, W.A. (eds.) Ways of Knowing in HCI, pp. 349–372. Springer, New York (2014). https://doi.org/10.1007/978-1-4939-0378-8_14
Splunk. https://www.splunk.com/
Fluentd. https://www.fluentd.org/
Loggly. https://www.loggly.com/
Logstach. https://www.elastic.co/
Graylog. https://www.graylog.org/
He, P., et al.: Towards automated log parsing for large-scale log data analysis. IEEE Trans. Dependable Secure Comput. 15, 931–944 (2018)
Surwase, V.: REST API modeling languages - a developer’s perspective (2016)
Acknowledgment
This work was supported in part by the National Science and Technology Major Project under Grant 2018ZX03001016; Engineering Research Center of Information Networks, Ministry of Education.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Haihong, E., Chen, Y., Song, M., Sun, M. (2020). Distributed Cloud Monitoring Platform Based on Log In-Sight. In: Zhang, X., Liu, G., Qiu, M., Xiang, W., Huang, T. (eds) Cloud Computing, Smart Grid and Innovative Frontiers in Telecommunications. CloudComp SmartGift 2019 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 322. Springer, Cham. https://doi.org/10.1007/978-3-030-48513-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-48513-9_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-48512-2
Online ISBN: 978-3-030-48513-9
eBook Packages: Computer ScienceComputer Science (R0)