Skip to main content
SpringerLink
Log in

Exploring the effect of training-time randomness on the performance of deep neural networks for intrusion detection

  • Data analytics and machine learning
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

The number of papers on machine learning and deep neural networks applied to intrusion detection systems (IDS) is ever-increasing. Differently from existing work on the topic, this paper explores the effect of training-time randomness of deep neural networks, which is overlooked by the related literature. Training-time randomness is regulated by the seed of the pseudorandom number generator, and affects the performance of IDS models. The seed selection is studied in conjunction with other critical learning parameters: to the best of our knowledge, there are no similar studies in IDS. The experiments are done with a recent and widely consolidated intrusion detection benchmark, which is used to train and test a neural network under different combinations of seeds and parameters both in supervised and semi-supervised learning modes. The results are inferred by a mixture of explorative analysis, design of experiments, and analysis of variance. According to the results, the choice of the seed yields either excellent or scarce detection metrics; more importantly, the seed selection might be as relevant as the other major learning parameters assessed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Availability of data and materials

The datasets and materials used and/or analyzed during the current study are available at the webpages reported in the manuscript.

Notes

  1. A flow record—often informally called network flow—holds the values of categorical and numerical features that provide context data and summary statistics computed from the packets pertaining to a network flow between a source computer and a destination across a network.

  2. https://keras.io/keras_tuner/.

  3. https://github.com/maxpumperla/hyperas.

  4. https://github.com/ahlashkari/CICFlowMeter.

  5. https://downloads.distrinet-research.be/WTMC2021/tools_datasets.html.

  6. https://stackoverflow.com/questions/75850086.

  7. https://github.com/NVIDIA/framework-determinism.

  8. https://developer.nvidia.com/blog/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/.

  9. https://www.tensorflow.org/guide/keras/serialization_and_saving.

References

  • Ahmad Z, Shahid Khan A, Wai Shiang C, Abdullah J, Ahmad F (2021) Network intrusion detection system: a systematic study of machine learning and deep learning approaches. Trans Emerg Telecommun Technol 32(1):e4150

    Article  Google Scholar 

  • Andresini G, Appice A, Caforio FP, Malerba D, Vessio G (2022) Roulette: a neural attention multi-output model for explainable network intrusion detection. Expert Syst Appl 201:117144

    Article  Google Scholar 

  • Aoudni Y, Donald C, Farouk A, Sahay KB, Babu DV, Tripathi V, Dhabliya D (2022) Cloud security based attack detection using transductive learning integrated with hidden Markov model. Pattern Recogn Lett 157:16–26

    Article  Google Scholar 

  • Apruzzese G, Pajola L, Conti M (2022) The cross-evaluation of machine learning-based network intrusion detection systems. IEEE Trans Netw Serv Manag 19(4):5152–5169

    Article  Google Scholar 

  • Atefinia R, Ahmadi M (2021) Network intrusion detection using multi-architectural modular deep neural network. J Supercomput 77(4):3571–3593

    Article  Google Scholar 

  • Bårli EM, Yazidi A, Viedma EH, Haugerud H (2021) DoS and DDoS mitigation using variational autoencoders. Comput Netw 199:108399

    Article  Google Scholar 

  • Catillo M, Pecchia A, Rak M, Villano U (2021) Demystifying the role of public intrusion datasets: a replication study of DoS network traffic data. Comput Secur 108:102341

    Article  Google Scholar 

  • Catillo M, Del Vecchio A, Pecchia A, Villano U (2022) Transferability of machine learning models learned from public intrusion detection datasets: the CICIDS2017 case study. Softw Qual J 30:955–981

  • Catillo M, Pecchia A, Villano U (2023a) CPS-GUARD: intrusion detection for cyber-physical systems and IoT devices using outlier-aware deep autoencoders. Comput Secur 129:103210

  • Catillo M, Pecchia A, Villano U (2023b) Successful intrusion detection with a single deep autoencoder: theory and practice. Softw Qual J 2023:1

  • Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv 41(3):15

    Article  Google Scholar 

  • de Carvalho Bertoli G, Junior Alves Pereira L, Saotome O, dos Santos AL (2023) Generalizing intrusion detection for heterogeneous networks: a stacked-unsupervised federated learning approach. Comput Secur 127:103106

    Article  Google Scholar 

  • Dina AS, Manivannan D (2021) Intrusion detection based on machine learning techniques in computer networks. Internet Things 16:100462

    Article  Google Scholar 

  • Engelen G, Rimmer V, Joosen W (2021) Troubleshooting an intrusion detection dataset: the CICIDS2017 case study. In: Proceedings of the security and privacy workshops. IEEE, New York, pp 7–12

  • Fellicious C, Weissgerber T, Granitzer M (2020) Effects of random seeds on the accuracy of convolutional neural networks. In: Machine learning, optimization, and data science. Springer, London, pp 93–102

  • Fisher R (1929) Tests of significance in harmonic analysis. Proc R Soc Lond 125:54–59

    Google Scholar 

  • Folino F, Folino G, Guarascio M, Pisani F, Pontieri L (2021) On learning effective ensembles of deep neural networks for intrusion detection. Inf Fusion 72:48–69

    Article  Google Scholar 

  • Gowdhaman V, Dhanapal R (2022) An intrusion detection system for wireless sensor networks using deep neural network. Soft Comput 26:13059–13067

    Article  Google Scholar 

  • He K, Kim DD, Asghar MR (2023) Adversarial machine learning for network intrusion detection systems: a comprehensive survey. IEEE Commun Surv Tutor 25(1):538–566

    Article  Google Scholar 

  • Imran M, Khan S, Hlavacs H et al (2022) Intrusion detection in networks using cuckoo search optimization. Soft Comput 26:10651–10663

    Article  Google Scholar 

  • Izmailov P, Podoprikhin D, Garipov T, Vetrov DP, Wilson AG (2018) Averaging weights leads to wider optima and better generalization. In: Proceedings of the conference on uncertainty in artificial intelligence

  • Jain R (1991) The art of computer systems performance analysis. Wiley, New York

    Google Scholar 

  • Kocher G, Kumar G (2021) Machine learning and deep learning methods for intrusion detection systems: recent developments and challenges. Soft Comput 25:9731–9763

    Article  Google Scholar 

  • Liao L, Li H, Shang W, Ma L (2022) An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks. ACM Trans Softw Eng Methodol 31(3):1

    Article  Google Scholar 

  • Maciá-Fernández G, Camacho J, Magán-Carrión R, García-Teodoro P, Therón R (2017) UGR’16: a new dataset for the evaluation of cyclostationarity-based network IDSs. Comput Secur 73:411–424

    Article  Google Scholar 

  • Madhyastha PS, Batra D (2019) On model stability as a function of random seed. In: CoNLL, pp 929–939

  • Moustafa N, Slay J (2015) UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). In: Proceedings of the military communications and information systems conference, pp 1–6

  • Sharafaldin I, Lashkari AH, Ghorbani AA (2018) Toward generating a new intrusion detection dataset and intrusion traffic characterization. In: Proceedings of the international conference on information systems security and privacy. SciTePress, pp 108–116

  • Shenfield A, Day D, Ayesh A (2018) Intelligent intrusion detection systems using artificial neural networks. ICT Express 4(2):95–99

    Article  Google Scholar 

  • Verkerken M, D’Hooge L, Wauters T, Volckaert B, De Turck F (2021) Towards model generalization for intrusion detection: unsupervised machine learning techniques. J Netw Syst Manag 30(1):12

    Article  Google Scholar 

  • Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11:3371–3408

    MathSciNet  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic, London

    Book  Google Scholar 

  • Zhang L, Lu X, Chen Z, Liu T, Chen Q, Li Z (2022) Adaptive deep learning for network intrusion detection by risk analysis. Neurocomputing 493:46–58

    Article  Google Scholar 

  • Zoppi T, Ceccarelli A (2021) Prepare for trouble and make it double! Supervised–unsupervised stacking for anomaly-based intrusion detection. J Netw Comput Appl 189:103106

    Article  Google Scholar 

Download references

Acknowledgements

Catillo acknowledges the Italian “PRIN 2020” project EMELIOT “Engineered MachinE Learning-intensive IoT systems”.

Funding

The authors declare no funding to report.

Author information

Author notes

  1. Marta Catillo, Antonio Pecchia, and Umberto Villano have contributed equally to this work.

Authors and Affiliations

  1. Dipartimento di Ingegneria, Università degli Studi del Sannio, Palazzo Bosco Lucarelli C.so Garibaldi 107, 82100, Benevento, Italy

    Marta Catillo, Antonio Pecchia & Umberto Villano

Authors
  1. Marta Catillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonio Pecchia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Umberto Villano
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MC, AP, and UV have contributed equally to this work.

Corresponding author

Correspondence to Marta Catillo.

Ethics declarations

Conflict of interest

The authors have no conflict of interest, and no financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Catillo, M., Pecchia, A. & Villano, U. Exploring the effect of training-time randomness on the performance of deep neural networks for intrusion detection. Soft Comput 28, 1957–1969 (2024). https://doi.org/10.1007/s00500-023-09552-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-023-09552-4

Keywords

Search

Navigation