Skip to main content

AEGR: a simple approach to gradient reversal in autoencoders for network anomaly detection

Abstract

Anomaly detection is referred to a process in which the aim is to detect data points that follow a different pattern from the majority of data points. With the rapid development of computer technology, protecting networks from various threats such as network intruders is becoming crucial. Traditional anomaly detection methods suffer from several well-known challenges that hinder their performance, such as the curse of high dimensionality. Autoencoders are unsupervised neural networks that have been used for the purpose of reducing dimensionality and also detecting network anomalies in large datasets. The performance of autoencoders debilitates when the training set contains noise and anomalies. In this paper, a new gradient-reversal method is proposed to overcome the influence of anomalies on the training phase for the purpose of detecting network anomalies. The method is different from other approaches as it does not require an anomaly-free training set and is based on reconstruction error. Once latent variables are extracted from the network, local outlier factor is used to separate normal data points from anomalies. A simple pruning approach and data augmentation are also added to improve performance further. The experimental results show that the proposed model can outperform other well-known approaches.

This is a preview of subscription content, access via your institution.

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Arzoo M, Neeraj K (2019) Deep learning models for traffic flow prediction in autonomous vehicles: a review, solutions, and challenges. Vehicular Commun 20:100184

    Article  Google Scholar 

  2. Aygun RC, Yavuz AG (2017) Network anomaly detection with stochastically improved autoencoder based models. In: 2017 IEEE 4th international conference on cyber security and cloud computing (CSCloud), pp 193–198. IEEE

  3. Bottou L (1991) Stochastic gradient learning in neural networks. Proc Neuro-Nımes 91(8):12

    Google Scholar 

  4. Breunig MM, Kriegel HP, Ng RT, Sander J (2000) LOF: identifying density-based local outliers. SIGMOD Rec., 29(2):93–104, may

  5. Breuniq MM, Kriegel HP, Ng RT, Sander J (2000) LOF: identifying density-based local outliers. SIGMOD Record (ACM Special Interest Group on Management of Data) 29(2):93–104

    Google Scholar 

  6. Cao VL, Nicolau M, McDermott J (2019) Learning neural representations for network anomaly detection. IEEE Trans Cybernet 49(8):3074–3087

    Article  Google Scholar 

  7. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv 41(3):15:1–15:58

    Article  Google Scholar 

  8. Charte D, Charte F, García S, del Jesus MJ, Herrera F (2018) A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines. Information Fusion 44:78–96

    Article  Google Scholar 

  9. Chen J, Sathe S, Aggarwal C, Turaga D (2017) Outlier detection with autoencoder ensembles. In: Proceedings of the 2017 SIAM international conference on data mining, pp 90–98. SIAM

  10. Chen Z, Yeo CK, Lee BS, Lau CT(2018) Autoencoder-based network anomaly detection. In: 2018 wireless telecommunications symposium (WTS), pp. 1–5

  11. Chow JK, Su Z, Wu J, Tan PS, Mao X, Wang YH (2020) Anomaly detection of defects on concrete structures with the convolutional autoencoder. Adv Eng Inf 45:101105

    Article  Google Scholar 

  12. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7(Jan):1–30

    MathSciNet  MATH  Google Scholar 

  13. Dheeru D, Taniskidou EK (2017) UCI machine learning repository

  14. Erfani SM, Rajasegarar S, Karunasekera S, Leckie C (2016) High-dimensional and large-scale anomaly detection using a linear one-class svm with deep learning. Pattern Recogn 58:121–134

    Article  Google Scholar 

  15. Ganin Y, Lempitsky V (2014) Unsupervised domain adaptation by backpropagation. arXiv:1409.7495

  16. García S, Grill M, Stiborek J, Zunino A (2014) An empirical comparison of botnet detection methods. Comput Secur 45:100–123

    Article  Google Scholar 

  17. Garg S, Kaur K, Kumar N, Rodrigues JJPC (2019) Hybrid deep-learning-based anomaly detection scheme for suspicious flow detection in SDN: a social multimedia perspective. IEEE Trans Multimedia 21(3):566–578

    Article  Google Scholar 

  18. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    MathSciNet  Article  Google Scholar 

  19. Hodge Victoria J, Jim A (2004) A survey of outlier detection methodologies. Artif Intell Rev 22(2):85–126

    Article  Google Scholar 

  20. Kerby DS (2014) The simple difference formula: an approach to teaching nonparametric correlation. Comprehens Psychol 3:11–IT

    Article  Google Scholar 

  21. Khan Shehroz S, Madden Michael G (2010) A survey of recent trends in one class classification. In: Coyle L, Freyne J (eds) Artificial intelligence and cognitive science: 20th irish conference, AICS 2009, Dublin, Ireland, August 19–21, 2009, Revised Selected Papers. Springer, Berlin, pp 188–197

    Chapter  Google Scholar 

  22. Lian J, Jia W, Zareapoor M, Zheng Y, Luo R, Jain DK, Kumar N (2020) Deep-learning-based small surface defect detection via an exaggerated local variation-based generative adversarial network. IEEE Trans Ind Inf 16(2):1343–1351

    Article  Google Scholar 

  23. Ma Y, Zhang P, Cao Y, Guo L (2013) Parallel auto-encoder for efficient outlier detection. In: Big Data, 2013 IEEE international conference on, pp 15–17. IEEE

  24. Moustafa N, Slay J (2015) UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). In: 2015 Military communications and information systems conference (MilCIS), pp 1–6, nov

  25. Munir M, Siddiqui SA, Dengel A, Ahmed S (2019) DeepAnT: a deep learning approach for unsupervised anomaly detection in time series. IEEE Access 7:1991–2005

    Article  Google Scholar 

  26. Nguyen HD, Tran KP, Thomassey S, Hamad M (2021) Forecasting and anomaly detection approaches using lstm and lstm autoencoder techniques with the applications in supply chain management. Int J Inf Manag 57:102282

    Article  Google Scholar 

  27. Provost F, Fawcett T (1997) Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. In: Proceedings of the third international conference on knowledge discovery and data mining, KDD’97, pp 43–48. AAAI Press

  28. Qi Y, Wang Y, Zheng X, Wu Z (2014) Robust feature learning by stacked autoencoder with maximum correntropy criterion. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 6716–6720. IEEE

  29. Safara F, Souri A, Serrizadeh M (2020) Improved intrusion detection method for communication networks using association rule mining and artificial neural networks. IET Commun 14(7):1192–1197

    Article  Google Scholar 

  30. Sakurada M, Yairi T (2014) Anomaly detection using autoencoders with nonlinear dimensionality reduction. In: Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis, p 4. ACM

  31. Schreyer M, Sattarov T, Borth D, Dengel A, Reimer B (2017) Detection of anomalies in large scale accounting data using deep autoencoder networks. arXiv:1709.05254

  32. Sun Z, Sun H (2019) Stacked denoising autoencoder with density-grid based clustering method for detecting outlier of wind turbine components. IEEE Access 7:13078–13091

  33. Sun J, Wang X, Xiong N, Shao J (2018) Learning sparse representation with variational auto-encoder for anomaly detection. IEEE Access 6:33353–33361

    Article  Google Scholar 

  34. Su S, Xiao L, Zhang Z, Gu F, Ruan L, Li S, He Z, Huo Z, Yan B, Wang H, et al (2017) N2DLOF: a new local density-based outlier detection approach for scattered data. In: High performance computing and communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), 2017 IEEE 19th International Conference on, pp 458–465. IEEE

  35. Tang B, He H (2017) A local density-based approach for outlier detection. Neurocomputing 241:171–180

    Article  Google Scholar 

  36. Tavallaee M, Bagheri E, Lu W, Ghorbani AA (2009) A detailed analysis of the KDD CUP 99 data set. In: 2009 IEEE symposium on computational intelligence for security and defense applications, pp 1–6. IEEE

  37. Van Cao L, Nicolau M, McDermott J (2016) A hybrid autoencoder and density estimation model for anomaly detection. In: Handl J, Hart E, Lewis PR, López-Ibáñez M, Ochoa G, Paechter B (eds) Parallel problem solving from nature–PPSN XIV. Springer, Cham, pp 717–726

    Chapter  Google Scholar 

  38. Wang Y, Yao H, Zhao S (2016) Auto-encoder based dimensionality reduction. Neurocomputing 184:232–242

    Article  Google Scholar 

  39. Yousefi-Azar M, Varadharajan V, Hamey L, Tupakula U (2017) Autoencoder-based feature learning for cyber security applications. In: 2017 International joint conference on neural networks (IJCNN), pp 3854–3861. IEEE

  40. Zhang Q, Yang LT, Chen Z, Li P (2018) A survey on deep learning for big data. Inf Fusion 42:146–157

    Article  Google Scholar 

  41. Zhou C, Paffenroth RC (2017) Anomaly detection with robust deep autoencoders. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 665–674. ACM

Download references

Funding

The authors acknowledge that this study has not been funded.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Zhi Yuan Chen.

Ethics declarations

Conflict of interest

All authors have participated in every stage of the research including but not limited to experimenting, analysis, drafting, reviewing and revising the work. All authors declare that they have no conflicts of interest.

Research involving human participants and/or animals

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Babaei, K., Chen, Z.Y. & Maul, T. AEGR: a simple approach to gradient reversal in autoencoders for network anomaly detection. Soft Comput 25, 15269–15280 (2021). https://doi.org/10.1007/s00500-021-06110-8

Download citation

Keywords

  • Network anomaly detection
  • High dimensionality
  • Autoencoders (AEs)
  • Local outlier factor (LOF)
  • Gradient reversal