Skip to main content
SpringerLink
Log in

Supervised Performance Anomaly Detection in HPC Data Centers

  • Conference paper
  • First Online:
Book cover The International Conference on Advanced Machine Learning Technologies and Applications (AMLTA2019) (AMLTA 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 921))

Abstract

High Performance Computing (HPC) systems play an important role in advancing scientific research due to a significant demand for processing power and speed grows. In practice, HPC systems are in the spot of interest of different businesses which account on this growing technology. The growing complexity of the HPC systems made it exposed to a great range of performance anomalies. Permanent management of such systems health has a huge impact financially and operationally. Several machine learning techniques can be used to identify these performance anomalies in such complex systems. This study compares the most commonly used three supervised machine learning algorithms for anomaly detection. We had applied these algorithms on the Fundación Pública Galega Centro Tecnolóxico de Supercomputación de Galicia (CESGA) memcpy metrics which is a benchmark used to measure memory performance for each CPU socket. Our study shows that Neural Network algorithm had the highest accuracy (93%), KNN algorithm had the highest value of precision (0.97), Gaussian Anomaly Detection algorithm had the highest value of recall (0. 99), and Neural Network algorithm had the highest value of F-measure (0.96).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Tuncer, O., et al.: Diagnosing performance variations in HPC applications using machine learning. In: International Supercomputing Conference. Springer (2017)

    Google Scholar 

  2. Sorkunlu, N., Chandola, V., Patra, A.: Tracking system behavior from resource usage data. In: IEEE International Conference on Cluster Computing (CLUSTER). IEEE (2017)

    Google Scholar 

  3. Garg, S.K., Versteeg, S., Buyya, R.: SMICloud: a framework for comparing and ranking cloud services. In: Fourth IEEE International Conference on Utility and Cloud Computing (UCC). IEEE (2011)

    Google Scholar 

  4. Kumar, N., Agarwal, S.: QoS based cloud service provider selection framework. Res. J. Recent Sci. (2014). ISSN 2277-2502

    Google Scholar 

  5. Ibidunmoye, O., Hernández-Rodriguez, F., Elmroth, E.: Performance anomaly detection and bottleneck identification. ACM Comput. Surv. (CSUR) 48(1), 4 (2015)

    Article  Google Scholar 

  6. Shipmon, D.T., et al.: Time series anomaly detection; detection of anomalous drops with limited features and sparse examples in noisy highly periodic data. arXiv preprint arXiv:1708.03665 (2017)

  7. Ahmad, S., Purdy, S.: Real-time anomaly detection for streaming analytics. arXiv preprint arXiv:1607.02480 (2016)

  8. Peiris, M., et al.: PAD: performance anomaly detection in multi-server distributed systems. In: IEEE 7th International Conference on Cloud Computing (CLOUD). IEEE (2014)

    Google Scholar 

  9. Cesga. http://www.cesga.es/en/. Accessed 16 Oct 2018

  10. Seabold, S., Perktold, J.: Proceedings of the 9th Python in Science Conference (2010)

    Google Scholar 

  11. Walt, S.v.d., Colbert, S.C., Varoquaux, G.: The NumPy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13(2), 22–30 (2011)

    Article  Google Scholar 

  12. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–28309 (2011)

    MathSciNet  MATH  Google Scholar 

  13. Shyu, M.-L., et al.: A novel anomaly detection scheme based on principal component classifier. Miami Univ Coral Gables Fl Dept of Electrical and Computer Engineering (2003)

    Google Scholar 

  14. Thill, M., Konen, W., Bäck, T.: Online anomaly detection on the webscope S5 dataset: a comparative study. In: Evolving and Adaptive Intelligent Systems (EAIS). IEEE (2017)

    Google Scholar 

  15. Panchal, G., Panchal, M.: Review on methods of selecting number of hidden nodes in artificial neural network. Int. J. Comput. Sci. Mob. Comput. 3(11), 455–464 (2014)

    MathSciNet  Google Scholar 

  16. Agrawal, S., Agrawal, J.: Survey on anomaly detection using data mining techniques. Procedia Comput. Sci. 60, 708–713 (2015)

    Article  Google Scholar 

  17. Hodo, E., et al.: Shallow and deep networks intrusion detection system: a taxonomy and survey. arXiv preprint arXiv:1701.02145 (2017)

Download references

Acknowledgments

The European Regional Development Fund (ERDF) and the Galician Regional Government under the agreement for funding the Atlantic Research Center for Information and Communication Technologies (AtlantTIC), the Spanish Ministry of Economy and Competitiveness under the National Science Program (TEC2014-54335-C4-3-R and TEC2017-84197-C4-2-R). Finally, the authors would like to thank the Supercomputing Center of Galicia (CESGA) for their support and resources in this research.

Author information

Authors and Affiliations

  1. Information System Department, Arab Academy for Science Technology and Maritime Transport, Cairo, Egypt

    Mohamed Soliman Halawa

  2. Information & Computing Lab., AtlantTIC Research Center School of Telecommunications Engineering, University of Vigo, Vigo, Spain

    Rebeca P. Díaz Redondo & Ana Fernández Vilas

Authors
  1. Mohamed Soliman Halawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rebeca P. Díaz Redondo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ana Fernández Vilas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohamed Soliman Halawa .

Editor information

Editors and Affiliations

  1. Faculty of Computers and Information, Cairo University, Giza, Egypt

    Aboul Ella Hassanien

  2. Faculty of Computers and Information, Benha University, Benha, Egypt

    Ahmad Taher Azar

  3. School of Computing, Science and Engineering, University of Salford, Salford, Greater Manchester, UK

    Tarek Gaber

  4. Department of Computer Science and Engineering, School of Computing and IT, Faculty of Engineering, Manipal University Jaipur, Jaipur, Rajasthan, India

    Roheet Bhatnagar

  5. Faculty of Computer and Information Science, Ain Shams University, Cairo, Egypt

    Mohamed F. Tolba

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Halawa, M.S., Díaz Redondo, R.P., Vilas, A.F. (2020). Supervised Performance Anomaly Detection in HPC Data Centers. In: Hassanien, A., Azar, A., Gaber, T., Bhatnagar, R., F. Tolba, M. (eds) The International Conference on Advanced Machine Learning Technologies and Applications (AMLTA2019). AMLTA 2019. Advances in Intelligent Systems and Computing, vol 921. Springer, Cham. https://doi.org/10.1007/978-3-030-14118-9_67

Download citation

Publish with us

Policies and ethics

Search

Navigation