Anomaly detection in cloud environment using artificial intelligence techniques

Abstract

Now days the usage of cloud environment is rapidly increasing in all the fields to run applications in virtual machines instead of physical hardware based machine. This increases the service availability and also reduces the cost. The usage of openstack cloud environment is also increasing both in academics and industry as it provides open source cloud services to run the application both for research and for production environment. One of the challenges in cloud environment is that the detection and prediction of the anomalies before they occur. In the traditional approach, the anomalies are detected manually by keeping track of threshold level and heartbeat. The recent research is happening on using machine learning techniques in detecting the anomalies before they occur. In this paper, we propose a model for anomaly detection in openstack cloud environment. In the proposed model, we used Stacked and Bidirectional LSTM models to build the neural network. For the experiment the data is collected from openstack using collectd. The collected data sets 10 features and class label. Using LSTM neural network, we were able to detect the anomalies in openstack environment. The proposed model achieved the detection accuracy of 94.61% for training set and 93.98% for the test set using binary cross entropy function as a loss function.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. 1.

    Bondan L, Wauters T, Volckaert B, De Turck F, Granville LZ (2017) Anomaly detection framework for SFC integrity in NFV environments. In: 2017 IEEE conference on network softwarization (NetSoft), Bologna, pp. 1–5. https://doi.org/10.1109/NETSOFT.2017.8004204

  2. 2.

    Girish L, Rao SKN (2016) Mathematical tools and methods for analysis of SDN: a comprehensive survey. In 2nd international conference on contemporary computing and informatics (IC3I), Noida, pp 718–724. https://doi.org/10.1109/IC3I.2016.7918055

  3. 3.

    Kourtis M, Xilouris G, Gardikis G, Koutras I (2016) Statistical-based anomaly detection for NFV services. In: 2016 IEEE conference on network function virtualization and software defined networks (NFV-SDN), Palo Alto, CA, pp 161–166. https://doi.org/10.1109/NFV-SDN.2016.7919492

  4. 4.

    Jitendra K, Rimsha G, Kumar SA (2018) Long short term memory recurrent neural network (LSTM-RNN) based workload forecasting model for cloud datacenters, Procedia computer science, vol 125. ISSN 676-682:1877-0509

  5. 5.

    Tran N, Nguyen T, Binh MN, Nguyen G (2018) A multivariate fuzzy time series resource forecast model for clouds using LSTM and data correlation analysis. Elsevier, Science Direct

  6. 6.

    Das A, Mueller F, Siegel C, Vishnu A (2018) Desh: deep learning for system health prediction of lead times to failure in HPC. ACM

  7. 7.

    Du M, Li F, Zheng G, Srikumar V (2017) DeepLog: anomaly detection and diagnosis from system logs through deep learning. ACM

  8. 8.

    Huch F, Golagha M, Petrovska A, Krauss A. Machine learning-based run-time anomaly detection in software systems: an industrial evaluation

  9. 9.

    Zhang C, Song D, Chen Y, Feng X, Lumezanu C, Cheng W, Ni J, Zong B, Chen H, Chawla NV (2018) A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. arxiv.1811.08055v1 [cs.LG]

  10. 10.

    Tang X (2019) Large-scale computing systems workload prediction using parallel improved LSTM neural network. IEEE Access 7:40525–40533. https://doi.org/10.1109/ACCESS.2019.2905634

    Article  Google Scholar 

  11. 11.

    Tang X (2019) Large-scale computing systems workload prediction using parallel improved LSTM neural network. IEEE Access

  12. 12.

    Radford B, Apolonio L, Trias A, Simpson J (2018) Network traffic anomaly detection using recurrent neural networks. arXiv:1803.10769

  13. 13.

    Girish L (2019) Efficient monitoring of time series data using dynamic alerting. IndiaRxiv. https://doi.org/10.26634/jcom.6.2.14870

  14. 14.

    Rashmi TV, Prasanna MK, Girish L (2015) Load balancing as a service in Openstack-Liberty. Int J Sci Technol Res 4(8):70–73

    Google Scholar 

  15. 15.

    Girish L, Rao SKN (2020) Quantifying sensitivity and performance degradation of virtual machines using machine learning. J Comput Theor Nanosci. https://doi.org/10.1166/jctn.2020.9019

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to L. Girish.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Girish, L., Rao, S.K.N. Anomaly detection in cloud environment using artificial intelligence techniques. Computing (2021). https://doi.org/10.1007/s00607-021-00941-x

Download citation

Keywords

  • Cloud computing
  • Recurrent neural network
  • Anomaly
  • LSTM
  • Openstack
  • InfluxDB
  • Artificial intelligence