Abstract
As the number of nodes in cluster systems continues to grow, leveraging scalable algorithms in all aspects of such systems becomes key to maintaining performance. While scalable algorithms have been applied successfully in some areas of parallel I/O, many operations are still performed in an uncoordinated manner. In this work we consider, in three file system scenarios, the possibilities for applying scalable algorithms to the many operations that make up the MPI-IO interface. From this evaluation we extract a set of file system characteristics that aid in developing scalable MPI-IO implementations.
This work was supported by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy, under Contract W-31-109-Eng-38.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
ALC, the ASCI Linux Cluster, http://www.llnl.gov/linux/alc/
Ching, A., Choudhary, A., Coloma, K., Liao, W.k., Ross, R., Gropp, W.: Noncontiguous I/O accesses through MPI-IO. In: Proceedings of the Third IEEE/ACM International Symposium on Cluster Computing and the Grid, Tokyo, Japan, May 2003, pp. 104–111. IEEE Computer Society Press, Los Alamitos (2003)
Ching, A., Choudhary, A., Liao, W.k., Ross, R., Gropp, W.: Noncontiguous I/O through PVFS. In: Proceedings of the 2002 IEEE International Conference on Cluster Computing (September 2002)
Ching, A., Choudhary, W., Liao, R.: Efficient structured data access in parallel file systems. In: Proceedings of Cluster 2003, Hong Kong (November 2003)
IBM DataStar Cluster, http://www.npaci.edu/DataStar/
IEEE/ANSI Std. 1003.1. Portable operating system interface (POSIX)–part 1: System application program interface (API) [C language], 1996 edition
Isaila, F., Tichy, W.F.: View I/O: Improving the performance of noncontiguous I/O. In: Proceedings of IEEE Cluster Computing Conference, Hong Kong (December 2003)
LCRC, the Argonne National Laboratory Computing Project, http://www.lcrc.anl.gov
Ma, X., Winslett, M., Lee, J., Yu, S.: Improving MPI IO output performance with active buffering plus threads. In: Proceedings of the International Parallel and Distributed Processing Symposium, April 2003, IEEE Computer Society Press, Los Alamitos (2003)
MPI-2: Extensions to the message-passing interface. The MPI Forum (July 1997)
Prost, J.-P., Treumann, R., Hedges, R., Jia, B., Koniges, A.: MPI-IO GPFS, an optimized implementation of MPI-IO on top of GPFS. In: Proceedings of Supercomputing 2001 (November 2001)
The Parallel Virtual File System, version 2, http://www.pvfs.org/pvfs2
Thakur, R., Choudhary, A.: An Extended Two-Phase Method for Accessing Sections of Out-of-Core Arrays. Scientific Programming 5(4), 301–317 (1996)
Thakur, R., Gropp, W., Lusk, E.: A case for using MPI’s derived datatypes to improve I/O performance. In: Proceedings of SC98: High Performance Networking and Computing, November 1998, ACM Press, New York (1998)
Worringen, J., Traff, J.L., Ritzdorf, H.: Fast parallel noncontiguous file access. In: Proceedings of SC 2003: High Performance Networking and Computing, Phoenix, AZ, November 2003, IEEE Computer Society Press, Los Alamitos (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Latham, R., Ross, R., Thakur, R. (2004). The Impact of File Systems on MPI-IO Scalability. In: Kranzlmüller, D., Kacsuk, P., Dongarra, J. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2004. Lecture Notes in Computer Science, vol 3241. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30218-6_18
Download citation
DOI: https://doi.org/10.1007/978-3-540-30218-6_18
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23163-9
Online ISBN: 978-3-540-30218-6
eBook Packages: Springer Book Archive