Abstract
Big data analytic needs a reliable data processing platform, which usually consists of large amount of distributed monitored objects, sometimes geographically dispersed ones. The rapidly increasing scale and complexity of a big data processing platform are making autonomous monitoring and management become much more crucial than before. In this paper, we design and implement an autonomous monitoring system - CMCloud to deal with these challenges faced by current big data processing platform. By introducing sequential flow control for multi-step operations of an action, CMCloud implements autonomous interaction between monitoring server and monitored objects as well as automatic fault diagnosis and recovery. CMCloud can be deployed into a big data processing platform to find, locate and process potential system faults timely and precisely, and then enhance the reliability of it.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
EMC Education Services: Cloud Infrastructure and Services. EMC Corporation, Hopkinton Massachusetts (2014)
Kutare, M., Eisenhauer, G., Wand, C., Schwan, K., Talwar, V., Wolf, M.: Online monitoring and analytics for managing large scale data centers. In: Proceedings of the 7th International Conference on Autonomic Computing, pp. 141–150. ACM, US (2010)
Liang, J., Ko, S.Y., Gupta, I., Nahrstedt, K.: MON: on-demand overlays for distributed system management. In: WORLDS 2005: Second Workshop on Real, Large Distributed Systems, pp. 13–18. USENIX, San Francisco (2005)
PlanetLab Consortium. https://www.planet-lab.org/. Accessed 2017
Zabbix LLC. https://www.zabbix.com/. Accessed 2018
Wu, Z.: Zabbix Enterprise Distributed Monitoring System. Mechanical Industry Press, Beijing (2014)
Ganglia Community. http://ganglia.info/. Accessed 07 Mar 2018
Nagios Enterprises: https://www.nagios.org/. Accessed 2018
Zhang, X.Y., Chen, G.S.: Intelligent monitoring system on cloud computing platform based on Ganglia and Nagios. J. Anhui Univ. Sci. Technol. (Nat. Sci.) 36(4), 69–74 (2016)
Fan, Z.: An active management framework for automatic fault detection and elimination in distributed systems. ICIC Express Lett. 2(1), 31–35 (2011)
Python Software Foundation. https://www.python.org/. Accessed 2018
Wikipedia. https://en.wikipedia.org/wiki/LAMP (software_bundle). Accessed 2018
Django Software Foundation. https://www.djangoproject.com/. Accessed 2018
Ben-Kiki, O., Evans, C., döt Net, I.: http://www.yaml.org/. Accessed 2018
Acknowledgment
This work is partially supported by the Special Fund for Basic Scientific Research of Central Colleges, Chang’An University (CHD2011TD009). The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the presentation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Fan, Z., Xu, M., Xi, J., Li, D. (2019). CMCloud: An Autonomous Monitoring System for Big Data Processing Platform. In: Li, J., Meng, X., Zhang, Y., Cui, W., Du, Z. (eds) Big Scientific Data Management. BigSDM 2018. Lecture Notes in Computer Science(), vol 11473. Springer, Cham. https://doi.org/10.1007/978-3-030-28061-1_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-28061-1_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28060-4
Online ISBN: 978-3-030-28061-1
eBook Packages: Computer ScienceComputer Science (R0)