Skip to main content
SpringerLink
Log in

SFC-NIDS: a sustainable and explainable flow filtering based concept drift-driven security approach for network introspection

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The evolving behavior of the attacks may affect the decision boundaries of the trained machine learning models. The issue has not been well investigated, especially with hypervisor-based security solutions where virtual machine (VM)’s network artifacts are introspected and analyzed. In this paper, we proposed a sustainable and explainable flow-filtering-based concept drift-driven network intrusion detection approach, called ‘SFC-NIDS’ which introspects network activities by analyzing VM traffic profile. The VM traffic is captured and pre-processed at the hypervisor to extract important network artifacts. The redundant and trivial network flows have been filtered using the proposed gradient descent-based flow filtering mechanism and validated using explainability. SFC-NIDS employs auto-encoders to reconstruct the traffic features to capture additional patterns. Afterward, the 1D-convolution neural network has been employed to learn and detect malicious attack flows. The model’s sustainability is ensured by integrating the drift detection mechanism with the decision model to retrain it with evolving attack patterns. The approach has been validated with virtual network traffic artifacts collected at the hypervisor and provides 98.9% accuracy, 99.03%, and F1-Score. In addition, the approach has also been validated using the KDD99 dataset, showcasing an accuracy of 99.97% and an F1-Score of 99.98%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

The data will be made available on request.

Notes

  1. https://www.theverge.com/2023/10/10/23911186/ddos-http2-vulnerability-blocked-amazon-aws-cloudflare-google-cloud.

  2. https://www.businesstoday.in/magazine/deep-dive/story/the-dark-side-of-the-cloud-how-cloud-is-becoming-prey-to-sophisticated-forms-of-cyber-attack-393051-2023-08-08.

  3. https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.

  4. https://www.unb.ca/cic/datasets/nsl.html.

  5. https://research.unsw.edu.au/projects/unsw-nb15-dataset.

  6. http://www.itoc.usma.edu/research/dataset/.

  7. https://libvirt.org/news.html.

  8. https://hackage.haskell.org/package/xenstore.

  9. https://github.com/ahlashkari/CICFlowMeter.

  10. https://riverml.xyz/dev/api/drift/ADWIN/.

  11. https://virusshare.com/.

  12. https://www.virustotal.com/gui/home/upload.

  13. https://drakvuf.com/.

  14. https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html

  15. https://www.cisa.gov/news-events/alerts/2014/06/02/gameover-zeus-p2p-malware

  16. https://www.itworldcanada.com/article/remcos-trojan-back-on-checks-points-top-10-list-of-global-threats/532402.

  17. https://shap.readthedocs.io/en/latest/generated/shap.plots.force.html

  18. https://www.cisa.gov/news-events/alerts/2018/07/20/emotet-malware

References

  1. Ahmad, Z., Shahid Khan, A., Wai Shiang, C., Abdullah, J., Ahmad, F.: Network intrusion detection system: a systematic study of machine learning and deep learning approaches. Trans. Emerg. Telecommun. Technol. 32(1), 1–12 (2021)

    Google Scholar 

  2. Du, J., Yang, K., Hu, Y., Jiang, L.: NIDS-CNNLSTM: network intrusion detection classification model based on deep learning. IEEE Access 11, 24808–24821 (2023)

    Article  Google Scholar 

  3. Mishra, P., Varadharajan, V., Tupakula, U., Pilli, E.S.: A detailed investigation and analysis of using machine learning techniques for intrusion detection. IEEE Commun. Surv. Tutor. 21(1), 686–728 (2018)

    Article  Google Scholar 

  4. Srinivas, B., Mandal, I., Keshavarao, S.: Virtual machine migration-based intrusion detection system in cloud environment using deep recurrent neural network. Cybern. Syst. (2022). https://doi.org/10.1080/01969722.2022.2122008

    Article  Google Scholar 

  5. Sakr, M.M., Tawfeeq, M.A., El-Sisi, A.B.: Network intrusion detection system based PSO-SVM for cloud computing. Int. J. Comput. Netw. Inf. Secur. 11(3), 22–29 (2019)

    Google Scholar 

  6. Leon, R.S., Kiperberg, M., Leon Zabag, A.A., Zaidenberg, N.J.: Hypervisor-assisted dynamic malware analysis. Cybersecurity 4(1), 1–14 (2021)

    Article  Google Scholar 

  7. Mishra, P., Pilli, E.S., Varadharajan, V., Tupakula, U.: Out-VM monitoring for malicious network packet detection in cloud. In: 2017 ISEA Asia Security and Privacy Conference, ISEASP 2017, Surat, India, pp. 1–10 (2017)

  8. Mittal, A., Dua, M.: Static-dynamic features and hybrid deep learning models based spoof detection system for ASV. Complex Intell. Syst. 8(2), 1153–1166 (2022)

    Article  Google Scholar 

  9. Muallem, A., Shetty, S., Pan, J., Zhao, J., Biswal, B.: Hoeffding tree algorithms for anomaly detection in streaming HTBPS: a survey. J. Inf. Secur. 08, 339–361 (2017)

    Google Scholar 

  10. Seraphim, I., Eswaran, P.: Analysis on intrusion detection system using machine learning techniques. In: Computer Networks, Big Data and IoT, pp. 423–441. Springer, Singapore (2021)

  11. Pradheep, D., Gokul, R., Naveen, V., Vijayarani, J.: Anomaly intrusion detection based on concept drift. Glob. J. Comput. Sci. Technol. 20(E2), 1–12 (2020)

    Google Scholar 

  12. Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., Zhang, G.: Learning under concept drift: a review. IEEE Trans. Knowl. Data Eng. 31(12), 2346–2363 (2018)

    Google Scholar 

  13. Ashiku, L., Dagli, C.: Network intrusion detection system using deep learning. Procedia Comput. Sci. 185, 239–247 (2021)

    Article  Google Scholar 

  14. Kumar, D., Pateriya, R.K., Gupta, R.K., Dehalwar, V., Sharma, A.: Ddos detection using deep learning. Procedia Comput. Sci. 218, 2420–2429 (2023)

    Article  Google Scholar 

  15. Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M., Inman, D.J.: 1d convolutional neural networks and applications: a survey. Mech. Syst. Signal Process. 151, 1–21 (2021)

    Article  Google Scholar 

  16. Bifet, A., Gavaldà, R.: Learning from time-changing data with adaptive windowing. In: 7th SIAM International Conference on Data Mining, vol. 7, pp. 1–17 (2007)

  17. Seth, S., Singh, G., Chahal, K.: Drift-based approach for evolving data stream classification in intrusion detection system. In: WCNC-2021: Workshop on Computer Networks & Communications, Chennai, India pp. 23–30 (2021)

  18. Shaji, N.S., Muthalagu, R., Pawar, P.M.: SD-IIDS: intelligent intrusion detection system for software-defined networks. Multimedia Tools Appl 83(4), 11077–11109 (2023)

    Article  Google Scholar 

  19. Priya, S., Uthra, R.A.: Deep learning framework for handling concept drift and class imbalanced complex decision-making on streaming data. Complex Intell. Syst. 9, 3499–3515 (2021)

    Article  Google Scholar 

  20. Yuan, X., Wang, R., Zhuang, Y., Zhu, K., Hao, J.: A concept drift based ensemble incremental learning approach for intrusion detection. In: 2018 IEEE International Conference on Internet of Things (IThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), pp. 350–357. IEEE (2018)

  21. Andresini, G., Appice, A., Loglisci, C., Belvedere, V., Redavid, D., Malerba, D.: A Network Intrusion Detection System for Concept Drifting Network Traffic Data, pp. 111–121. Springer, Cham (2021)

  22. Kuppa, A., Le-Khac, N.-A.: Learn to adapt: Robust drift detection in security domain. Comput. Electr. Eng. 102, 1–13 (2022)

    Article  Google Scholar 

  23. Andresini, G., Pendlebury, F., Pierazzi, F., Loglisci, C., Appice, A., Cavallaro, L.: SOMNIA: towards concept-drift robustness in network intrusion detection. In: Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security (AISec), pp. 111–122. ACM, New York (2021)

  24. Nautiyal, A., Saklani, S., Pant, A., Agarwal, Y., Gaur, A., Mishra, P.: VNSecure: an explainable virtual network attack detection. In: IC3-2023: Proceedings of the 2023 Fifteenth International Conference on Contemporary Computing, pp. 153–160 (2023)

  25. Horchulhack, P., Viegas, E.K., Lopez, M.A.: A stream learning intrusion detection system for concept drifting network traffic. In: 2022 6th Cyber Security in Networking Conference (CSNet), pp. 1–7. IEEE (2022)

  26. Jain, M., Kaur, G.: Distributed anomaly detection using concept drift detection based hybrid ensemble techniques in streamed network data. Clust. Comput. 24, 2099–2114 (2021)

    Article  Google Scholar 

  27. Patil, R., Dudeja, H., Modi, C.: Designing an efficient security framework for detecting intrusions in virtual network of cloud computing. Comput. Secur. 85, 402–422 (2019)

    Article  Google Scholar 

  28. Rajeswari, P.V.N., Shashi, M., Rao, T.K., Rajya Lakshmi, M., Kiran, L.V.: Effective intrusion detection system using concept drifting data stream and support vector machine. Concurr. Comput. Pract. Exp. 34(21), 7118 (2022)

    Article  Google Scholar 

  29. Hinder, F., Vaquet, V., Brinkrolf, J., Hammer, B.: Model-based explanations of concept drift. Neurocomputing 555, 126640 (2023). https://doi.org/10.1016/j.neucom.2023.126640

    Article  Google Scholar 

  30. Liu, W., Zhu, C., Ding, Z., Zhang, H., Liu, Q.: Multiclass imbalanced and concept drift network traffic classification framework based on online active learning. Eng. Appl. Artif. Intell. 117, 105607 (2023). https://doi.org/10.1016/j.engappai.2022.105607

    Article  Google Scholar 

  31. Coppolino, L., D’Antonio, S., Nardone, R., Romano, L.: A self-adaptation-based approach to resilience improvement of complex internets of utility systems. Environ. Syst. Decis. 3, 708–720 (2023)

    Article  Google Scholar 

  32. Wankhade, K.K., Jondhale, K.C., Dongre, S.S.: A clustering and ensemble based classifier for data stream classification. Appl. Soft Comput. 102, 107076 (2021)

    Article  Google Scholar 

  33. Jain, M., Kaur, G., Saxena, V.: A k-means clustering and svm based hybrid concept drift detection technique for network anomaly detection. Expert Syst. Appl. 193, 1–18 (2022)

    Article  Google Scholar 

  34. Chisnall, D.: The Definitive Guide to the Xen Hypervisor, 1st edn. Prentice Hall Press, Upper Saddle (2007)

    Google Scholar 

  35. Zhang, C., Yao, M., Chen, W., Zhang, S., Chen, D., Wu, Y.: Gradient descent optimization in deep learning model training based on multistage and method combination strategy. Secur. Commun. Netw. 2021, 1–15 (2021)

    Google Scholar 

  36. Alzubaidi, L., Zhang, J., Humaidi, A.J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M.A., Al-Amidie, M., Farhan, L.: Review of deep learning: concepts, cnn architectures, challenges, applications, future directions. J. Big Data 8, 1–74 (2021)

    Article  Google Scholar 

  37. Bottou, L.: Stochastic gradient descent tricks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade, pp. 421–436. Springer, Berlin (2012)

  38. Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint (2016). arXiv:1609.04747

  39. Bank, D., Koenigstein, N., Giryes, R.: Autoencoders. Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook, pp. 353–374. Springer, Berlin (2023)

  40. Sarhan, M., Layeghy, S., Portmann, M.: Evaluating standard feature sets towards increased generalisability and explainability of ml-based network intrusion detection. Big Data Res. 30(C), 1–9 (2022)

    Google Scholar 

  41. Santos, C.F.G.D., Papa, J.P.: Avoiding overfitting: a survey on regularization methods for convolutional neural networks. ACM Comput. Surv. (CSUR) 54(10s), 1–25 (2022)

    Article  Google Scholar 

  42. Ghosh, S., Das, N., Nasipuri, M.: Reshaping inputs for convolutional neural network: Some common and uncommon methods. Pattern Recogn. 93, 79–94 (2019)

    Article  Google Scholar 

  43. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint, pp. 1–9 (2014). arXiv:1412.6806

  44. Agrahari, S., Singh, A.K.: Concept drift detection in data stream mining: a literature review. J. King Saud Univ. Comput. Inf. Sci. 34(10, Part B), 9523–9540 (2022)

    Google Scholar 

  45. Huang, D.T.J., Koh, Y.S., Dobbie, G., Bifet, A.: Drift detection using stream volatility. In: Machine Learning and Knowledge Discovery in Databases, pp. 417–432. Springer, Cham (2015)

  46. Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17, pp. 4768–4777. Curran Associates, Red Hook (2017)

  47. Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you? explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)

  48. Joshi, L.M., Kumar, M., Bharti, R.: Understanding threats in hypervisor, its forensics mechanism and its research challenges. Int. J. Comput. Appl. 119(1), 1–5 (2015)

    Google Scholar 

  49. Deylami, H., Gutierrez, J., Sinha, R.: Kororā: a secure live virtual machine job migration framework for cloud systems integrity. Array 19, 100312 (2023)

    Article  Google Scholar 

  50. Ortega-Fernandez, I., Sestelo, M., Burguillo, J.C., Pinon-Blanco, C.: Network intrusion detection system for DDoS attacks in ICS using deep autoencoders. Wirel. Netw. (2023). https://doi.org/10.1007/s11276-022-03214-3

    Article  Google Scholar 

  51. Elmasry, W., Akbulut, A., Zaim, A.H.: A design of an integrated cloud-based intrusion detection system with third party cloud service. Open Comput. Sci. 11(1), 365–379 (2021)

    Article  Google Scholar 

  52. Arun kumar, M., Ashok kumar, K.: Malicious attack detection approach in cloud computing using machine learning techniques. Soft. Comput. 26(23), 13097–13107 (2022)

    Article  Google Scholar 

  53. Shlens, J.: Notes on kullback-leibler divergence and likelihood. arXiv preprint (2014). arXiv:1404.2000

Download references

Acknowledgements

The authors would like to express their gratitude to the Science and Engineering Research Board, Department of Science and Technology (SERB-DST) for their intellectual generosity and research assistance.

Funding

This work is supported by SERB-POWER Grant [File No. SPG/2021/002003] and SERB-POWER Mobility Grant [File No. SPM/2022/000004] under Science and Engineering Research Board, Department of Science and Technology (SERB-DST), Govt. of India.

Author information

Authors and Affiliations

  1. Department of Computer Science, School of Technology, Doon University, Dehradun, Uttarakhand, India

    Arjun Singh, Preeti Mishra & Avantika Gaur

  2. Department of Mathematics, University of Padua, Padova, Italy

    P. Vinod & Mauro Conti

Authors
  1. Arjun Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Preeti Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. Vinod
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Avantika Gaur
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mauro Conti
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Arjun Singh: Software, validation, investigation, data curation, writing—-original draft, writing—review and editing. Preeti Mishra: Conceptualization, investigation, supervision, writing—original draft, writing—review and editing. Vinod P.: Conceptualization, investigation, supervision, writing—original draft, writing—review and editing. Avantika Gaur: Software, validation, investigation, writing—original draft, writing—review and editing. Mauro Conti: Supervision, review and editing.

Corresponding author

Correspondence to Preeti Mishra.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, A., Mishra, P., Vinod, P. et al. SFC-NIDS: a sustainable and explainable flow filtering based concept drift-driven security approach for network introspection. Cluster Comput (2024). https://doi.org/10.1007/s10586-024-04444-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10586-024-04444-0

Keywords

Search

Navigation