Skip to main content
SpringerLink
Log in

Anomaly detection framework for highly scattered and dynamic data on large-scale networks using AWS

  • Original Research
  • Published:
International Journal of Information Technology Aims and scope Submit manuscript

Abstract

Network anomaly detection is crucial for securing computer networks and communications. However, handling highly scattered data in large-scale, dynamic networks poses challenges, leading to increased false positives and false negatives in anomaly identification. To address this, a new anomaly detection framework employing ensemble learning techniques was introduced. This framework uses decision trees for scattered data and gradient-boosting algorithms for accuracy in dynamic network behaviors. It successfully identifies known and unknown abnormalities while ensuring scalability and reducing false positives and negatives. The framework was tested on CICIDS2017, NSLKDD, and KDDCUP99 datasets, achieving outstanding accuracy rates of 100% on CICIDS2017 and KDDCUP99 and 99.7% on NSLKDD. Deployed on Amazon Web Services, it accurately detected anomalies in new data inputs. Comparative analysis against existing models highlighted the framework's superiority in detecting anomalies in highly scattered data within complex network behaviors. Its performance measures demonstrate its effectiveness as a leading solution in network anomaly detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and materials

Datasets are available on Kaggle.

References

  1. Haider U, Waqas M, Hanif M, Alasmary H, Qaisar SM (2023) Network load prediction and anomaly detection using ensemble learning in 5G cellular networks. Comput Commun 197:141–150

    Article  Google Scholar 

  2. Singh O, Singh KK (2023) An approach to classify lung and colon cancer of histopathology images using deep feature extraction and an ensemble method. Int J Inform Technol. https://doi.org/10.1007/s41870-023-01487-1

    Article  Google Scholar 

  3. Yan F, Huang H, Pedrycz W, Hirota K (2023) Automated breast cancer detection in mammography using ensemble classifier and feature weighting algorithms. Expert Syst Appl 227:120282

    Article  Google Scholar 

  4. Wanda P (2023) GRUSpam: robust e-mail spam detection using gated recurrent unit (GRU) algorithm. Int J Inform Technol. https://doi.org/10.1007/s41870-023-01516-z

    Article  Google Scholar 

  5. Sudar KM, Deepalakshmi P, Singh A, Srinivasu PN (2023) TFAD: TCP flooding attack detection in software-defined networking using proxy-based and machine learning-based mechanisms. Clust Comput 26(2):1461–1477

    Article  Google Scholar 

  6. Wang X, Wang Y, Javaheri Z, Almutairi L, Moghadamnejad N, Younes OS (2023) Federated deep learning for anomaly detection in the internet of things. Comput Electr Eng 108:108651

    Article  Google Scholar 

  7. Sarkar A, Sharma HS, Singh MM (2023) A supervised machine learning-based solution for efficient network intrusion detection using ensemble learning based on hyperparameter optimization. Int J Inf Technol 15(1):423–434

    Google Scholar 

  8. Xin R, Liu H, Chen P, Zhao Z (2023) Robust and accurate performance anomaly detection and prediction for cloud applications: a novel ensemble learning-based framework. J Cloud Comput 12(1):1–16

    Article  Google Scholar 

  9. Folino G, Otranto Godano C, Pisani FS (2023) An ensemble-based framework for user behaviour anomaly detection and classification for cybersecurity. J Supercomput. https://doi.org/10.1007/s11227-023-05049-x

    Article  Google Scholar 

  10. Muhammad AR, Sukarno P, Wardana AA (2023) Integrated security information and event management (SIEM) with intrusion detection system (IDS) for live analysis based on machine learning. Proced Comput Sci 217:1406–1415

    Article  Google Scholar 

  11. Logeswari G, Bose S, Anitha T (2023) An intrusion detection system for sdn using machine learning. Intell Autom Soft Comput 35(1):867–880

    Article  Google Scholar 

  12. Al-Shareeda MA, Manickam S, Saare MA (2023) DDoS attacks detection using machine learning and deep learning techniques: analysis and comparison. Bull Electr Eng Inform 12(2):930–939

    Article  Google Scholar 

  13. Rawat R, Garg B, Pachlasiya K, Mahor V, Telang S, Chouhan M, Mishra R (2022) SCNTA: monitoring of network availability and activity for identification of anomalies using machine learning approaches. Int J Inform Technol Web Eng (IJITWE) 17(1):1–19

    Google Scholar 

  14. Cerdà-Alabern L, Iuhasz G, Gemmi G (2023) Anomaly detection for fault detection in wireless community networks using machine learning. Comput Commun 202:191–203

    Article  Google Scholar 

  15. Shaeiri Z, Kazemitabar SJ (2020) Fast unsupervised automobile insurance fraud detection based on spectral ranking of anomalies. Int J Eng 33(7):1240–1248

    Google Scholar 

  16. Yadav A, Kohli N (2021) Prolong stability period in node pairing protocol for wireless sensor networks. Int J Eng 34(12):2679–2687

    Google Scholar 

  17. Gupta PK, Singh NK, Mahajan V (2021) Intrusion detection in cyber-physical layer of smart grid using intelligent loop based artificial neural network technique. Int J Eng 34(5):1250–1256

    Google Scholar 

  18. Singh R (2022) "Performance optimization of autoencoder neural network based model for anomaly detection in network traffic." 2022 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE). IEEE

  19. Hasan M, Islam MM, Zarif MII, Hashem MMA (2019) Attack and anomaly detection in IoT sensors in IoT sites using machine learning approaches. Internet Things 7:100059

    Article  Google Scholar 

  20. Hamamoto AH, Carvalho LF, Sampaio LDH, Abrão T, Proença ML Jr (2018) Network anomaly detection system using genetic algorithm and fuzzy logic. Expert Syst Appl 92:390–402

    Article  Google Scholar 

  21. Kasim O (2020) An efficient and robust deep learning-based network anomaly detection against distributed denial of service attacks. Comput Netw 180:107390

    Article  Google Scholar 

  22. Muniyandi AP, Rajeswari R, Rajaram R (2012) Network anomaly detection by cascading k-Means clustering and C4. 5 decision tree algorithms. Proced Eng 30:174–182

    Article  Google Scholar 

  23. Aissa NB, Guerroumi M (2016) Semi-supervised statistical approach for network anomaly detection. Proced Comput Sci 83:1090–1095

    Article  Google Scholar 

  24. Bovenzi G, Aceto G, Ciuonzo D, Montieri A, Persico V, Pescapé A (2023) Network anomaly detection methods in IoT environments via deep learning: a Fair comparison of performance and robustness. Comput Secur 128:103167

    Article  Google Scholar 

  25. Vishwakarma M, Kesswani N (2023) A new two-phase intrusion detection system with Naïve Bayes machine learning for data classification and elliptic envelop method for anomaly detection. Decis Anal J 7:100233

    Article  Google Scholar 

  26. Yin Y, Jang-Jaccard J, Xu W, Singh A, Zhu J, Sabrina F, Kwak J (2023) IGRF-RFE: a hybrid feature selection method for MLP-based network intrusion detection on UNSW-NB15 dataset. Journal of Big Data 10(1):1–26

    Article  Google Scholar 

  27. Siddiqui T, Mustaqeem M (2023) Performance evaluation of software defect prediction with NASA dataset using machine learning techniques. Int J Inform Technol 15(8):4131–4139

    Google Scholar 

  28. Singh G, Thanaya I (2023) Predicting earnings per share using feature-engineered extreme gradient boosting models and constructing alpha trading strategies. Int J Inform Technol. https://doi.org/10.1007/s41870-023-01450-0

    Article  Google Scholar 

  29. Jintanachaiwat W, Siriborvornratanakul T (2023) Vision-based image similarity measurement for image search similarity. Int J Inform Technol. https://doi.org/10.1007/s41870-023-01437-x

    Article  Google Scholar 

  30. Guo D, Liu Z, Li R (2023). RegraphGAN: A graph generative adversarial network model for dynamic network anomaly detection. Neural Networks

  31. AWS. https://aws.amazon.com/sagemaker/

  32. Steven Huang, KDD-CUP99 https://www.kaggle.com/datasets/galaxyh/kdd-cup-1999-data

  33. M Hassan Zaib, NSL-KDD. https://www.kaggle.com/datasets/hassan06/nslkdd

  34. https://www.unb.ca/cic/datasets/ids-2017.html

  35. Singh R, Srivastav G (2021) Novel framework for anomaly detection using machine learning technique on CIC-IDS2017 Dataset. In 2021 International Conference on Technological Advancements and Innovations (ICTAI) (pp. 632–636). IEEE

  36. Singh R, Srivastava N, Kumar A (2021) Machine learning techniques for anomaly detection in network traffic. In 2021 Sixth International Conference on Image Information Processing (ICIIP) (Vol. 6, pp. 261–266). IEEE

Download references

Acknowledgements

I thank to Mr. Ashutosh Kumar for his valuable suggestions an guidance related to AWS services.

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Amity University, Lucknow, 226010, India

    Richa Singh & Nidhi Srivastava

  2. SRM Institute of Science and Technology, Ghaziabad, 201204, India

    Ashwani Kumar

Authors
  1. Richa Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nidhi Srivastava
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ashwani Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richa Singh.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

Not applicable.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, R., Srivastava, N. & Kumar, A. Anomaly detection framework for highly scattered and dynamic data on large-scale networks using AWS. Int. j. inf. tecnol. (2024). https://doi.org/10.1007/s41870-024-01765-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s41870-024-01765-6

Keywords

Search

Navigation