Abstract
Job schedulers on High Performance Computing systems can serve more purposes than just maximising computing resource utilisation if they are equipped with more awareness on other aspects of the system. In this work, we focus on making a job scheduler I/O-aware to assist system I/O management. We propose SchedP as the first practical effort on I/O-aware job scheduling that can work in production HPC environment. It trains neural network model to predict each job’s I/O pattern, then makes a delay decision if starting a job right away will lead to I/O congestion in the system. We integrate it into Slurm and performed evaluations with real HPC workloads in production environment for about a month. The results show: a) the neural network model of SchedP reached over 99% for both training and test accuracy on predicting jobs’ I/O patterns; b) SchedP has obvious effect on alleviating system I/O contention.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ahn, D.H., Garlick, J., Grondona, M., Lipari, D., Springmeyer, B., Schulz, M.: Flux: a next-generation resource management framework for large HPC centers. In: 2014 43rd International Conference on Parallel Processing Workshops, pp. 9–17. IEEE (2014)
Fan, Y.: Job scheduling in high performance computing. arXiv preprint arXiv:2109.09269 (2021)
Herbein, S., et al.: Scalable I/O-aware job scheduling for burst buffer enabled HPC clusters. In: Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing, pp. 69–80 (2016)
Li, D., Dong, M., Tang, Y., Ota, K.: A novel disk I/O scheduling framework of virtualized storage system. Clust. Comput. 22(1), 2395–2405 (2019)
Li, H., Liao, J., Liu, X.: Merging and prioritizing optimization in block I/O scheduling of disk storage. J. Circ. Syst. Comput. 30(10), 2150186 (2021)
Liu, J., Chen, Y., Zhuang, Y.: Hierarchical I/O scheduling for collective I/O. In: 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, pp. 211–218 (2013). https://doi.org/10.1109/CCGrid.2013.30
Liu, Y., Gunasekaran, R., Ma, X., Vazhkudai, S.S.: Server-side log data analytics for I/O workload characterization and coordination on large shared storage systems. In: SC2016: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 819–829. IEEE (2016)
McKenna, R., Gamblin, T., Moody, A., de Supinski, B., Taufer, M.: Forecasting storms in parallel file systems
Wyatt, M.R., Herbein, S., Gamblin, T., Moody, A., Ahn, D.H., Taufer, M.: PRIONN: predicting runtime and IO using neural networks. In: Proceedings of the 47th International Conference on Parallel Processing, pp. 1–12 (2018)
Wyatt II, M.R., Gamblin, T., Moody, A., Taufer, M.: Revealing the power of neural networks to capture accurate job resource usage from unparsed job scripts and application inputs (2017)
Zhou, Z., et al.: I/O-aware batch scheduling for petascale computing systems. In: 2015 IEEE International Conference on Cluster Computing, pp. 254–263 (2015). https://doi.org/10.1109/CLUSTER.2015.45
Acknowledgements
This work is supported by National Key R &D Programme of China under grant number 2018YFA0404603.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Wu, K., Wei, J., Lin, J. (2022). SchedP: I/O-aware Job Scheduling in Large-Scale Production HPC Systems. In: Liu, S., Wei, X. (eds) Network and Parallel Computing. NPC 2022. Lecture Notes in Computer Science, vol 13615. Springer, Cham. https://doi.org/10.1007/978-3-031-21395-3_29
Download citation
DOI: https://doi.org/10.1007/978-3-031-21395-3_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21394-6
Online ISBN: 978-3-031-21395-3
eBook Packages: Computer ScienceComputer Science (R0)