Abstract
Each software always has log data. Log data is essential for developers and end-users. With log data, we can indicate the information of system error, warning events or whether our system processes are loading successfully. This paper aims to provide a virtualization platform for our campus NetFlow log data in order user can use for further analysis. Our system used Ceph storage system and ELK Stack technology. Ceph provides a vast amount of scalable storage and ELK consists of three components, namely: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search engine in which can be used for searching all kinds of documents. Logstash is used for managing centralize, transform and stash our data to Elasticsearch and Kibana is used for visualizing Elasticsearch data and an excellent tool for a real-time data analytics and decision making. Our data is continuous data. Approximately our data growth up 38 GB per day and our data have grown more than 8TB until now.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Schroder, C.: Build a secure logging server with syslog (Part 2). Enterprise Networking Planet News, trends and advice for network managers. N.p. Web, 29 Nov 2010 (n.d.)
Rastogi, R., Akash, S., Shobha, G., Poonam, G., Pratiba, D., Singh, A.: Design and development of generic web based framework for log analysis. In: Region 10 Conference (TENCON), 2016 IEEE, pp. 232–236. IEEE (2016)
Awad, M., Menasc, D.A.: Automatic workload characterization using system log analysis. In: Computer Measurement Group Conference on Performance and Capacity, San Antonio, TX (2015)
Yang, C.-T., Chen, S.-T., Den, W. , Wang, Y.-T., Kristiani, E.: Implementation of an intelligent indoor environmental monitoring and management system in cloud. Future Gener. Comput. Syst. (2018)
Gormley, C., Tong, Z.: Elasticsearch: The Definitive Guide: A Distributed Real-Time Search and Analytics Engine. O’Reilly Media Inc., Sebastopol (2015)
Sanjappa, S., Ahmed, M.: Analysis of logs by using logstash. In: Proceedings of the 5th International Conference on Frontiers in Intelligent Computing: Theory and Applications, pp. 579–585. Springer, Singapore (2017)
Nginx introduction: http://www.webopedia.com/TERM/N/nginx.html
Kononenko, O., Baysal, O., Holmes, R., Godfrey, M.W.: Mining modern repositories with elasticsearch. In: Proceedings of the 11th Working Conference on Mining Software Repositories, pp. 328–331 (2014)
Yang, C.-T., Chen, C.-J., Chen, T.-Y.: Implementation of Ceph Storage with Big Data for Performance Comparison, pp. 625–633. Springer, Singapore (2017)
Zhan, K., Piao, A.H.: Optimization of ceph reads/writes based on multi-threaded algorithms. In: 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp. 719–725. IEEE (2016)
Open Nebula: https://opennebula.org/
Bagnasco, S., Berzano, D., Guarise, A., Lusso, S., Masera, M., Vallero, S.: Monitoring of IaaS and scientific applications on the Cloud using the Elasticsearch ecosystem. J. Phys. Conf. Ser. 608(1), 012016 (2015). (IOP Publishing)
Gupta, P., Nair, S.: Survey paper on elastic search. Int. J. Sci. Res. (IJSR) (2014)
Poat, M.D., Lauret, J.: Performance and advanced data placement techniques with Ceph’s distributed storage system. J. Phys. Conf. Ser. 762(1), 012025 (2016). (IOP Publishing)
Elasticsearch guidance page: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
Learn about Java Technology: https://java.com/en/about/
Apache http server: http://httpd.apache.org/
Appleyard, R., Adams, J.: Using the ELK Stack for CASTOR Application Logging at RAL. In: International Symposium on Grids and Clouds (ISGC), vol. 15, no. 20 (2015)
Prakash, T., Kakkar, M., Patel, K.: Geo-identification of web users through logs using ELK stack. In: 2016 6th International Conference on Cloud System and Big Data Engineering (Confluence), pp. 606–610. IEEE (2016)
Acknowledgements
This work was sponsored by the Ministry of Science and Technology, Taiwan (R.O.C.), under grant numbers MOST 104-2221-E-029-010-MY3 and MOST 106-3114-E-029-003.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, YT. et al. (2019). The Implementation of NetFlow Log System Using Ceph and ELK Stack. In: Hung, J., Yen, N., Hui, L. (eds) Frontier Computing. FC 2018. Lecture Notes in Electrical Engineering, vol 542. Springer, Singapore. https://doi.org/10.1007/978-981-13-3648-5_29
Download citation
DOI: https://doi.org/10.1007/978-981-13-3648-5_29
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-3647-8
Online ISBN: 978-981-13-3648-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)