Using XGBoost for Cyberattack Detection and Analysis in a Network Log System with ELK Stack

  • Cing-Han Lai
  • Chao-Tung YangEmail author
  • Endah Kristiani
  • Jung-Chun Liu
  • Yu-Wei Chan
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 551)


Recently, cyberattackers have been developing more sophisticated ways to attack systems. Accordingly, identifying these attacks is getting more complicated in time. On many situations, network administrators were not capable of recognizing these attacks effectively or respond quickly. Whereas, to monitor and analyze the network log data which is very large and complicated is challenging. Therefore, in this case, there is a need to use artificial intelligence and machine learning techniques. In this paper, we develop a monitoring and analysis system for network log data. First, we used Elasticsearch, Logstash, and Kibana (ELK Stack) to monitor the network system. Second, we analyze the network log data use ‘eXtreme Gradient Boosting’ (XGBoost) to build a model of attack event detections. Finally, we use the XGBoost model to do cross-validated with the ELK Stack.


Cyber security Machine Learning ELK Stack XGBoost NetFlow Log 



This work was sponsored by the Ministry of Science and Technology (MOST), Taiwan, under Grant No. 107-2221-E-029-008 and 107-2218-E-029-003.


  1. 1.
    Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system, 10 Jun 2016Google Scholar
  2. 2.
    Al-Qurishi, M., Alrubaian, M., Mizanur Rahman, Sk.Md., Alamri, A., Hassanab, M.M.: A prediction system of Sybil attack in social network using deep-regression model. Future Gener. Comput. Syst. 87, 743–753 (2018)Google Scholar
  3. 3.
    Zhang, J., Gardner, R., Vukotic, I.: Anomaly detection in wide area network meshes using two machine learning algorithms. Future Gener. Comput. Syst. 93, 418–426 (2019)CrossRefGoogle Scholar
  4. 4.
    Liu, H., Lang, B., Liu, M., Yan, H.: CNN and RNN based payload classification methods for attack detection. Knowl.-Based Syst. 163, 332–341 (2019)CrossRefGoogle Scholar
  5. 5.
    Prakash, T., Kakkar, M., Patel, K.: Geo identification of web users through logs using ELK stack. In: 6th International Conference - Cloud System and Big Data Engineering (2016)Google Scholar
  6. 6.
    Yang, C.-T., Chen, C.-J., Tsan, Y.-T., Liu, P.-Y., Chan, Y.-W., Chan, W.-C.: An implementation of real-time air quality and influenza-like illness data storage and processing platform. Comput. Hum. Behav. 100, 266–274 (2018)CrossRefGoogle Scholar
  7. 7.
    Kozik, R., Choras, M., Ficco, M., Palmieri, F.: A scalable distributed machine learning approach for attack detection in edge computing environments. J. Parallel Distrib. Comput. 119, 18–26 (2018)CrossRefGoogle Scholar
  8. 8.
    Sharafaldin, I., Habibi, A., Ali, L., Ghorbani, A.: An evaluation framework for network security visualizations. Comput. Secur. 84, 70–92 (2019)CrossRefGoogle Scholar
  9. 9.
    Yuan, X., Li, C., Li, X.: Deep defense identifying DDoS attack via deep learning. In: 2017 IEEE International Conference on Smart Computing (2017)Google Scholar
  10. 10.
    Diro, A.A., Chilamkurti, N.: Distributed attack detection scheme using deep learning approach for Internet of Things. Future Gener. Comput. Syst. 82, 761–768 (2018)CrossRefGoogle Scholar
  11. 11.
    Sahingoz, O.K., Buber, E., Demir, O., Diric, B.: Machine learning based phishing detection from URLs. Expert Syst. Appl. 117, 345–357 (2019)CrossRefGoogle Scholar
  12. 12.
    Sun, P., Li, J., Bhuiyan, M.Z.A., Wang, L., Li, B.: Modeling and clustering attacker activities in IoT through machine learning techniques. Inf. Sci. 479, 456–471 (2019)CrossRefGoogle Scholar
  13. 13.
    Bagnasco, S., Berzano, D., Guarise, A., Lusso, S., Masera, M., Vallero, S.: Monitoring of IaaS and scientific applications on the Cloud using the Elasticsearch ecosystem. In: Journal of Physics: Conference Series, vol. 608, pp. 012016. IOP Publishing (2015)Google Scholar
  14. 14.
    Peterson, P.: Unmasking deceptive attacks with machine learning. Comput. Fraud Secur. 2018(11), 15–17 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Cing-Han Lai
    • 1
  • Chao-Tung Yang
    • 1
    Email author
  • Endah Kristiani
    • 1
    • 2
    • 3
  • Jung-Chun Liu
    • 1
  • Yu-Wei Chan
    • 4
  1. 1.Department of Computer ScienceTunghai UniversityTaichung CityTaiwan (R.O.C.)
  2. 2.Department of Industrial Engineering and Enterprise InformationTunghai UniversityTaichung CityTaiwan (R.O.C.)
  3. 3.Department of Informatics, Faculty of Engineering and Computer ScienceKrida Wacana Christian UniversityJakartaIndonesia
  4. 4.College of Computing and InformaticsProvidence UniversityTaichung CityTaiwan (R.O.C.)

Personalised recommendations