Skip to main content
SpringerLink
Log in

ML-Based Methodology for HPC Facilities Supervision

  • Conference paper
  • First Online:
High Performance Computing (ISC High Performance 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13999))

Included in the following conference series:

  • 1015 Accesses

Abstract

Monitoring supercomputing facilities tends to be very time consuming and error-prone as the size of the collected data and the number of supervised devices do not cease to increase. In this paper, we propose a methodology to supervise those facilities based on measurements performed on devices at different levels of the infrastructure. Through its three phases -raw data cleaning, ML-based processing and visualisation using our developed tool- it facilitates the supervision of the computing center facilities and helps detecting irregular behaviours leading to manual correction actions. The case of the energy consumption is considered to illustrate the usefulness of this methodology and highlight its valuable results but it can be applied to any other target metric.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    CEA: “Commissariat à l’énergie atomique et aux énergies alternatives” for French Alternative Energies and Atomic Energy Commission.

  2. 2.

    Partnership for Advanced Computing in Europe.

  3. 3.

    Density-Based Spatial Clustering Applications with Noise.

  4. 4.

    Hierarchical Density-Based Spatial Clustering Applications with Noise.

  5. 5.

    Hierarchical Agglomerative Clustering.

References

  1. Agerwala, T.: Challenges on the road to exascale computing. In: 22nd Annual International Conference on Supercomputing (2008)

    Google Scholar 

  2. Alvin, K., Barrett, B., Brightwell, R., Dosanjih, S.S.: On the path to exascale. Int. J. Distrib. Syst. Technol. 1, 1–22 (2011)

    Article  Google Scholar 

  3. Bajal, E., Katara, V., Bhatia, M., Hooda, M.: A review of clustering algorithms: comparison of DBSCAN and K-mean with oversampling and t-SNE. J. Recent Patents Eng. 16(2), 17–31 (2022)

    Google Scholar 

  4. Bautista, E., Romanus, M., Davis, T., Whithney, C., Kubaska, T.: Collecting, monitoring, and analyzing facility and systems data at the national energy research scientific computing center. In: 48th International Conference on Parallel Processing (2019)

    Google Scholar 

  5. Borghesi, A., Bartolini, A., Lombardi, M., Milano, M., Benini, L.: Predictive modeling for job power consumption in HPC systems. In: Kunkel, J.M., Balaji, P., Dongarra, J. (eds.) ISC High Performance 2016. LNCS, vol. 9697, pp. 181–199. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41321-1_10

    Chapter  Google Scholar 

  6. Bourassa, N., Johnson, W., Broughton, J., Carter, D.M., Joy, S.: Operational data analytics: Optimizing the national energy research computing center cooling systems. In: 48th International Conference on parallel Processing, pp. 1–7 (2019)

    Google Scholar 

  7. Campello, R.J.G.B., Moulavi, D., Sander, J.: Density-based clustering based on hierarchical density estimates. In: Pei, J., Tseng, V.S., Cao, L., Motoda, H., Xu, G. (eds.) PAKDD 2013. LNCS (LNAI), vol. 7819, pp. 160–172. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37456-2_14

    Chapter  Google Scholar 

  8. Corbalan, J., Alonso, L., Aneas, J., Brochard, L.: Energy optimization and analysis with ear. In: IEEE International Conference on Cluster Computing (2020)

    Google Scholar 

  9. Dani, M.C., Doreau, H., Alt, S.: K-means application for anomaly detection and log classification in HPC. In: Benferhat, S., Tabia, K., Ali, M. (eds.) IEA/AIE 2017. LNCS (LNAI), vol. 10351, pp. 201–210. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60045-1_23

    Chapter  Google Scholar 

  10. Dash website (2023). https://dash.plotly.com/. Accessed 08 Mar 2023

  11. Gao, J., Zheng, F., Qi, F.: Sunway supercomputer architecture towards exascale computing: analysis and practice. China Inf. Sci. 64, 141101 (2021)

    Article  Google Scholar 

  12. van der Maaten, L., Hinton, G.: Viualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)

    MATH  Google Scholar 

  13. Molan, M., Borghesi, A., Cesarini, D., Benini, L., Bartolini, A.: RUAD: unsupervised anomaly detection in HPC systems. Future Gener. Comput. Syst. 141(C), 542–554 (2023)

    Article  Google Scholar 

  14. Ozer, G., Netti, A., Tafani, D., Schulz, M.: Characterizing HPC performance variation with monitoring and unsupervised learning. In: Jagode, H., Anzt, H., Juckeland, G., Ltaief, H. (eds.) ISC High Performance 2020. LNCS, vol. 12321, pp. 280–292. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59851-8_18

    Chapter  Google Scholar 

  15. Shalf, J., Dosanjh, S., Morrison, J.: Exascale computing technology challenges. In: 9th International Conference on High Performance Computing for Computational Science (2010)

    Google Scholar 

  16. Shoukourian, H., Wilde, T., Labrenz, D., Bode, A.: Using machine learning for data center cooling infrastructure efficiency prediction. In: 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) (2017)

    Google Scholar 

  17. Shrikant, K., Gupta, V., Khandare, A., Furia, P.: A comparative study of clustering algorithm. In: Balas, V.E., Semwal, V.B., Khandare, A. (eds.) Intelligent Computing and Networking. Lecture Notes in Networks and Systems, vol. 301, pp. 219–235. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-4863-2_19

    Chapter  Google Scholar 

  18. Singh, H.V., Girdhar, A., Dahiya, S.: A literature survey based on DBSCAN algorithms. In: 6th International Conference on Intelligent Computing and Control Systems, pp. 751–758 (2022)

    Google Scholar 

  19. Su, Y., Zhou, J., Ying, J., Zhou, M., Zhou, B.: Computing infrastructure construction and optimization for high-performance computing and artificial intelligence. CCF Trans. High Perform. Comput. 3(4), 331–343 (2021). https://doi.org/10.1007/s42514-021-00080-x

    Article  Google Scholar 

  20. Tanash, M., Dunn, B., Andresen, D., Hsu, W., Yang, H., Okanlawon, A.: Improving HPC system performance by predicting job resources via supervised machine learning. In: Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning). PEARC 2019 (2019)

    Google Scholar 

  21. Terai, M., Shoji, F., Tsukamoto, T., Yamochi, Y.: A study of operational impact on power usage effectiveness using facility metrics and server operation logs in the k computer. In: IEEE International Conference on Cluster Computing (2020)

    Google Scholar 

  22. Terai, M., Yamamoto, K., Miura, S., Shoji, F.: An operational data collecting and monitoring platform for Fugaku: system overviews and case studies in the prelaunch service period. In: Jagode, H., Anzt, H., Ltaief, H., Luszczek, P. (eds.) ISC High Performance 2021. LNCS, vol. 12761, pp. 365–377. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-90539-2_24

    Chapter  Google Scholar 

  23. TGCC-CEA. https://www-hpc.cea.fr/en/TGCC.html

  24. Top500 the list (2022). https://www.top500.org/lists/top500/2022/11/

Download references

Author information

Authors and Affiliations

  1. CEA, Bruyères-le-Châtel, France

    Laetitia Anton, Sophie Willemot, Sebastien Gougeaud & Soraya Zertal

  2. Li-PaRAD, University of Versailles (UVSQ), Guyancourt, France

    Soraya Zertal

Authors
  1. Laetitia Anton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sophie Willemot
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sebastien Gougeaud
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Soraya Zertal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sebastien Gougeaud .

Editor information

Editors and Affiliations

  1. University of New Mexico, Albuquerque, NM, USA

    Amanda Bienz

  2. University of Edinburgh, Edinburgh, UK

    Michèle Weiland

  3. Université Paris-Saclay, Gif sur Yvette, France

    Marc Baboulin

  4. CERFACS, Toulouse, France

    Carola Kruse

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Anton, L., Willemot, S., Gougeaud, S., Zertal, S. (2023). ML-Based Methodology for HPC Facilities Supervision. In: Bienz, A., Weiland, M., Baboulin, M., Kruse, C. (eds) High Performance Computing. ISC High Performance 2023. Lecture Notes in Computer Science, vol 13999. Springer, Cham. https://doi.org/10.1007/978-3-031-40843-4_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-40843-4_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-40842-7

  • Online ISBN: 978-3-031-40843-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation