Abstract
HPC system usually uses a shared filesystem like Lustre as temporary file storage like scratch directory. It needs automated purge process to remove unused files for maintaining optimal performance of the shared filesystem. However, the purge process in large capacity file system takes much time to search and remove target files. In this paper, accelerated purge processes using MPI are proposed. First, master/slave parallel purge (MSPP) process is the method that master node distributes purge tasks among slave nodes. Second, evenly distributed purge (EDPP) process is the method that all node are involved in purge process that improves load balancing. Experimental results show that the purging time of proposed EDPP method for 1958 GB scratch data has been reduced by 4.04 and 2.47 times, respectively, when it is compared with the results of single node purge (SNP) and MSPP methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Feiyi, W., Sarp, O., Galen, S., Oleg, D., Tom, W., Isaac, H.: Understanding lustre filesystem internals. Oak Ridge National Lab., Technical Report ORNL/TM-2009/117 (2009)
Jaehyun, H., Deoksang, K., Hyeonsang, E.: Improving the performance of lustre file system in HPC environments. In: IEEE International Workshops on Foundations and Applications of Self* Systems, pp. 84–89. IEEE (2016)
Hongzhang, S., John, S.: Using IOR to Analyze the I/O Performance for HPC Platforms. Lawrence Berkeley National Laboratory, Berkeley (2007)
Tiezhu, Z., Verdi, M., Shoubin, D., Simon, S.: Evaluation of a performance model of lustre file system. In: 2010 Fifth Annual ChinaGrid Conference (ChinaGrid), pp. 191–196. IEEE (2010)
Stephen, C.S., Gregory, G.P., Douglas, B.: Wide area filesystem performance using lustre on the teragrid. Technical Report (2007)
Gary, L.R., Jesse, H., Rick, M.: Data management practices on large-scale lustre scratch file systems. In: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment, p. 36. ACM (2014)
National Institute of Supercomputing and Networking, KISTI. http://www.nisn.re.kr
Acknowledgments
This research was supported by Korea Institute of Science and Technology Information (KISTI).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Kwon, MW., Yoon, J., Hong, T., Park, C. (2018). Accelerated Purge Processes of Parallel File System on HPC by Using MPI Programming. In: Park, J., Loia, V., Yi, G., Sung, Y. (eds) Advances in Computer Science and Ubiquitous Computing. CUTE CSA 2017 2017. Lecture Notes in Electrical Engineering, vol 474. Springer, Singapore. https://doi.org/10.1007/978-981-10-7605-3_181
Download citation
DOI: https://doi.org/10.1007/978-981-10-7605-3_181
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-7604-6
Online ISBN: 978-981-10-7605-3
eBook Packages: EngineeringEngineering (R0)