Skip to main content
SpringerLink
Log in

Big Data-Aware Intrusion Detection System in Communication Networks: a Deep Learning Approach

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

One of the most important parameters that hackers have always considered is obtaining information about the status of computer networks, such as hacking into databases and computer networks used in defense systems. Hence, these networks are always exposed to dangerous attacks. On the other hand, networks and hosts face a large amount of data every second. Hence, intrusion detection mechanisms have to mine this growing mountain of data for possible intrusive patterns from the security perspective. This environment and conditions make it hard to detect intrusions fast and accurately. Therefore, to identify such intrusions, it is necessary to design an intrusion detection system using big data techniques that can handle these types of data that have big data nature in detecting unauthorized access to a communication network. Therefore, this article employs a big data-aware deep learning method to design an efficient and effective Intrusion Detection System (IDS) to cope with these challenges. We designed a specific architecture of Long Short-Term Memory (LSTM), and this model can detect complex relationships and long-term dependencies between incoming traffic packets. Through this way, we could reduce the number of false alarms and increase the accuracy of the designed intrusion detection system. Moreover, using big data analytic techniques can improve the speed of deep learning algorithms in this paper, which have low execution speed due to their high complexity. Actually, using these techniques increases the speed of execution of our complex model. Our extensive experiments are on the BigDL directly on top of the Spark framework and train with the NSL-KDD dataset. Results show that the proposed algorithm, called BDL-IDS, outperforms other IDS schemes, such as traditional machine learning and Artificial Neural Network, in terms of detection rate (20%), false alarm rate (60%), accuracy (15%), and training time (70%).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

The data of this paper is the result of simulation and all the data are presented in the form of graphs inside the paper. There is no private data in this article.

References

  1. Padmasiri, M. A. T., Ganepola, V. V. V., Herath, R. K. H. M.S.D, Welagedara, L. P., & Vekneswaran, P. (2020). Survey on Deep learning based Network Intrusion Detection and Prevention Systems

  2. Hadi, M.S., Lawey, A.Q., El-Gorashi, T.E., Elmirghani, J.M.: Big data analytics for wireless and wired network design: a survey. Comput. Netw. 132, 180–199 (2018)

    Article  Google Scholar 

  3. Atli, B.G., Miche, Y., Kalliola, A., Oliver, I., Holtmanns, S., Lendasse, A.: Anomaly-based intrusion detection using extreme learning machine and aggregation of network traffic statistics in probability space. Cogn. Comput. 10(5), 848–863 (2018)

    Article  Google Scholar 

  4. Lin, W.C., Ke, S.W., Tsai, C.F.: CANN: an intrusion detection system based on combining cluster centers and nearest neighbors. Knowl.-Based Syst. 78, 13–21 (2015)

    Article  Google Scholar 

  5. Jha, J., Ragha, L.: Intrusion detection system using support vector machine. International Journal of Applied Information Systems (IJAIS). 3, 25–30 (2013)

    Google Scholar 

  6. Labonne, M., Olivereau, A., Polvé, B., & Zeghlache, D. (2019, January). A cascade-structured meta-specialists approach for neural network-based intrusion detection. In 2019 16th IEEE Annual Consumer Communications & Networking Conference (CCNC) (pp. 1-6). IEEE

  7. Man, J., & Sun, G. (2021). A Residual Learning-Based Network Intrusion Detection System. Security and Communication Networks, 2021

  8. Kannari, P. R., Shariff, N. C., & Biradar, R. L. (2021). Network intrusion detection using sparse autoencoder with swish-PReLU activation model. Journal of ambient intelligence and humanized computing, 1-13

  9. Ling, J., Zhu, Z., Luo, Y., Wang, H.: An intrusion detection method for industrial control systems based on bidirectional simple recurrent unit. Computers & Electrical Engineering. 91, 107049 (2021)

    Article  Google Scholar 

  10. Yang, J., Lim, H.: Deep learning approach for detecting malicious activities over encrypted secure channels. IEEE Access. 9, 39229–39244 (2021)

    Article  Google Scholar 

  11. Yan, S.: Understanding LSTM and its diagrams, ML review. https://blog.mlreview.com/understanding-lstm-and-its-diagrams-37e2f46f1714 (2016)

  12. Dai, J. J., Wang, Y., Qiu, X., Ding, D., Zhang, Y., Wang, Y., ... Song, G.: Bigdl: A distributed deep learning framework for big data. In Proceedings of the ACM Symposium on Cloud Computing, pp. 50–60

  13. Mishra, M., Kang, M., Woo, J.: Rating Prediction using Deep Learning and Spark. In The 11th International Conference on Internet (ICONI 2019), pp307–310. (2019)

  14. Yin, C., Zhu, Y., Fei, J., He, X.: A deep learning approach for intrusion detection using recurrent neural networks. Ieee Access. 5, 21954–21961 (2017)

    Article  Google Scholar 

  15. Yu, Y., Long, J., & Cai, Z. (2017). Network intrusion detection through stacking dilated convolutional autoencoders. Security and Communication Networks, 2017

  16. Putchala, Manoj Kumar. "Deep learning approach for intrusion detection system (ids) in the internet of things (iot) network using gated recurrent neural networks (gru)." (2017)

  17. Kim, J., Shin, N., Jo, S. Y., & Kim, S. H. (2017, February). Method of intrusion detection using deep neural network. In 2017 IEEE international conference on big data and smart computing (BigComp) (pp. 313-316). IEEE

  18. Fontugne, R., Mazel, J., & Fukuda, K. (2014). Hashdoop: a MapReduce framework for network anomaly detection. In 2014 IEEE conference on computer communications workshops (INFOCOM WKSHPS) (pp. 494-499). IEEE

  19. Bachupally, Y. R., Yuan, X., & Roy, K. (2016). Network security analysis using big data technology. In SoutheastCon 2016 (pp. 1-4). IEEE

  20. Holtz, M.D., David, B., de Sousa Júnior, R.T.: Building scalable distributed intrusion detection systems based on the mapreduce framework. Telecomunicacoes (Santa Rita do Sapucai). 13(2), 22–31 (2011)

    Google Scholar 

  21. Sharma, R., Sharma, P., Mishra, P., & Pilli, E. S. (2016). Towards MapReduce based classification approaches for intrusion detection. In 2016 6th international conference-cloud system and big data engineering (confluence) (pp. 361-367). IEEE

  22. Elayni, M., & Jemili, F. (2017, June). Using mongodb databases for training and combining intrusion detection datasets. In international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (pp. 17-29). Springer, Cham

  23. Kulariya, M., P. Saraf, R. Ranjan and G. P. Gupta (2016). Performance analysis of network intrusion detection schemes using apache spark. Communication and signal processing (ICCSP), 2016 international conference on, IEEE

  24. Zhong, W., Yu, N., Ai, C.: Applying big data based deep learning system to intrusion detection. Big Data Mining and Analytics. 3(3), 181–195 (2020)

    Article  Google Scholar 

  25. Shone, N., Ngoc, T.N., Phai, V.D., Shi, Q.: A deep learning approach to network intrusion detection. IEEE transactions on emerging topics in computational intelligence. 2(1), 41–50 (2018)

    Article  Google Scholar 

  26. Li, Z., Qin, Z., Huang, K., Yang, X., & Ye, S. (2017). Intrusion detection using convolutional neural networks for representation learning. In international conference on neural information processing (pp. 858-866). Springer, Cham

  27. Gouveia, A., & Correia, M. (2017, June). A systematic approach for the application of restricted Boltzmann machines in network intrusion detection. In international work-conference on artificial neural networks (pp. 432-446). Springer, Cham

  28. Althubiti, S. A., Jones, E. M., & Roy, K. (2018, November). Lstm for anomaly-based network intrusion detection. In 2018 28th international telecommunication networks and applications conference (ITNAC) (pp. 1-3). IEEE

  29. Kim, J., Kim, J., Thu, H. L. T., & Kim, H. (2016, February). Long short term memory recurrent neural network classifier for intrusion detection. In 2016 international conference on platform technology and service (PlatCon) (pp. 1-5). IEEE

  30. Boukhalfa, A., Abdellaoui, A., Hmina, N., Chaoui, H.: LSTM deep learning method for network intrusion detection system. International Journal of Electrical & Computer Engineering. (2088–8708), 10 (2020)

  31. Ahmad, Z., Shahid Khan, A., Wai Shiang, C., Abdullah, J., Ahmad, F.: Network intrusion detection system: a systematic study of machine learning and deep learning approaches. Trans. Emerg. Telecommun. Technol. 32(1), e4150 (2021)

    Google Scholar 

  32. Zarei, S. M., & Fotohi, R. (2021). Defense against flooding attacks using probabilistic thresholds in the internet of things ecosystem. Security and Privacy, e152

  33. Zaminkar, M., Sarkohaki, F., Fotohi, R.: A method based on encryption and node rating for securing the RPL protocol communications in the IoT ecosystem. Int. J. Commun. Syst. 34(3), e4693 (2021)

    Article  Google Scholar 

  34. Faraji-Biregani, M., & Fotohi, R. (2020). Secure communication between UAVs using a method based on smart agents in unmanned aerial vehicles. The journal of supercomputing, 1-28

  35. Gupta, A. S. G., Prasad, G. S., & Nayak, S. R. (2019). A New and Secure Intrusion Detecting System for Detection of Anomalies within the Big Data. In Cloud Computing for Geospatial Big Data Analytics (Pp. 177–190). Springer, Cham

  36. Zaminkar, M., Fotohi, R.: SoS-RPL: securing internet of things against sinkhole attack using RPL protocol-based node rating and ranking mechanism. Wirel. Pers. Commun. 114, 1287–1312 (2020)

    Article  Google Scholar 

  37. Özgür, A., Erdem, H.: A review of KDD99 dataset usage in intrusion detection and machine learning between 2010 and 2015. PeerJ Preprints, 4, e1954v1

  38. Atat, R., Liu, L., Wu, J., Li, G., Ye, C., Yang, Y.: Big data meet cyber-physical systems: a panoramic survey. IEEE Access. 6, 73603–73636 (2018)

    Article  Google Scholar 

  39. Mabodi, K., Yusefi, M., Zandiyan, S., Irankhah, L., & Fotohi, R. (2020). Multi-level trust-based intelligence schema for securing of internet of things (IoT) against security threats using cryptographic authentication. The journal of supercomputing, 1-26

  40. Sahani, R., Rout, C., Badajena, J. C., Jena, A. K., & Das, H. (2018). Classification of Intrusion Detection Using Data Mining Techniques. In Progress in Computing, Analytics and Networking (Pp. 753–764). Springer, Singapore

  41. Kim, J., Kim, J., Kim, H.: An approach to build an efficient intrusion detection classifier. Journal of Platform Technology. 3(4), 43–52 (2015)

    Google Scholar 

  42. Alrawashdeh, Khaled, and Carla Purdy. "Toward an online anomaly intrusion detection system based on deep learning." 2016 15th IEEE international conference on machine learning and applications (ICMLA). IEEE, 2016

  43. " NSL-KDD Datasets Research Canadian Institute for Cybersecurity UNB". [Online]. Available: https://www.unb.ca/cic/datasets/nsl.html. [Accessed: 26- May- 2021]

  44. Karataş, F., Korkmaz, S.A.: Big data: controlling fraud by using machine learning libraries on spark. International Journal of Applied Mathematics Electronics and Computers. 6(1), 1–5 (2018)

    Article  Google Scholar 

  45. Natesan, P., Rajalaxmi, R.R., Gowrison, G., Balasubramanie, P.: Hadoop based parallel binary bat algorithm for network intrusion detection. Int. J. Parallel Prog. 45(5), 1194–1213 (2017)

    Article  Google Scholar 

  46. Othman, S.M., Ba-Alwi, F.M., Alsohybe, N.T., Al-Hashida, A.Y.: Intrusion detection model using machine learning algorithm on big data environment. Journal of Big Data. 5(1), 1–12 (2018)

    Article  Google Scholar 

  47. Kulkarni, S. (2017). Design Approaches of Intrusion Detection Systems Using Ensembling Algorithms (Doctoral dissertation, Dublin, National College of Ireland)

Download references

Author information

Authors and Affiliations

  1. Computer Engineering Department, University of Mohaghegh Ardabili, Ardabil, Iran

    Mahzad Mahdavisharif & Shahram Jamali

  2. Center of Excellence in Future Networks, Iran University of Science and Technology, Tehran, Iran

    Shahram Jamali

  3. Faculty of Computer Science and Engineering, Shahid Beheshti University, G. C. Evin, Tehran, 1983969411, Iran

    Reza Fotohi

Authors
  1. Mahzad Mahdavisharif
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shahram Jamali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Reza Fotohi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Reza Fotohi.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahdavisharif, M., Jamali, S. & Fotohi, R. Big Data-Aware Intrusion Detection System in Communication Networks: a Deep Learning Approach. J Grid Computing 19, 46 (2021). https://doi.org/10.1007/s10723-021-09581-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10723-021-09581-z

Keywords

Search

Navigation