Advertisement

Overview of the MPI-IO Parallel I/O Interface

  • Peter Corbett
  • Dror Feitelson
  • Sam Fineberg
  • Yarsun Hsu
  • Bill Nitzberg
  • Jean-Pierre Prost
  • Marc Snirt
  • Bernard Traversat
  • Parkson Wong
Part of the The Kluwer International Series in Engineering and Computer Science book series (SECS, volume 362)

Abstract

Thanks to MPI, writing portable message passing parallel programs is almost a reality. One of the remaining problems is file I/O. Although parallel file systems support similar interfaces, the lack of a standard makes developing a truly portable program impossible. It is not feasible to develop large scientific applications from scratch for each generation of parallel machine, and, in the scientific world, a program is not considered truly portable unless it not only compiles, but also runs efficiently.

The MPI-IO interface is being proposed as an extension to the MPI standard to fill this need. MPI-IO supports a high-level interface to describe the partitioning of file data among processes, a collective interface describing complete transfers of global data structures between process memories and files, asynchronous I/O operations, allowing computation to be overlapped with I/O, and optimization of physical file layout on storage devices (disks).

Keywords

File System Data Layout Collective Operation Parallel File System File Pointer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Robert Bennett, Kelvin Bryant, Alan Sussman, Raja Das, and Joel Saltz. Jovian: A framework for optimizing parallel I/O. In Proceedings of the Scalable Parallel Libraries Conference, pages 10–20. IEEE Computer Society Press, October 1994.Google Scholar
  2. [2]
    Michael L. Best, Adam Greenberg, Craig Stanfill, and Lewis W. Tucker. CMMD I/O: A parallel Unix I/O. In Proceedings of the Seventh International Parallel Processing Symposium, pages 489–495, 1993.CrossRefGoogle Scholar
  3. [3]
    Alok Choudhary, Rajesh Bordawekar, Michael Harry, Rakesh Krishnaiyer, Ravi Ponnusamy, Tarvinder Singh, and Rajeev Thakur. PASSION: parallel and scalable software for input-output. Technical Report SCCS-636, ECE Dept., NPAC and CASE Center, Syracuse University, September 1994.Google Scholar
  4. [4a]
    Peter F. Corbett, Sandra Johnson Baylor, and Dror G. Feitelson. Overview of the Vesta parallel file system. In IPPS ’93 Workshop on Input/Output in Parallel Computer Systems, pages 1–16, 1993.Google Scholar
  5. [4b]
    Peter F. Corbett, Sandra Johnson Baylor, and Dror G. Feitelson. Also published in Computer Architecture News 21(5), December 1993, pages 7–14.CrossRefGoogle Scholar
  6. [5]
    Peter F. Corbett and Dror G. Feitelson. Design and implementation of the Vesta parallel file system. In Proceedings of the Scalable High-Performance Computing Conference, pages 63–70, 1994.CrossRefGoogle Scholar
  7. [6]
    Peter F. Corbett, Dror G. Feitelson, Jean-Pierre Prost, George S. Almasi, Sandra Johnson Baylor, Anthony S. Bolmarcich, Yarsun Hsu, Julian Satran, Marc Snir, Robert Colao, Brian Herr, Joseph Kavaky, Thomas R. Morgan, and Anthony Zlotek. Parallel file systems for the IBM SP computers. IBM Systems Journal, pages 222–248, 1995.Google Scholar
  8. [7]
    Peter F. Corbett, Dror G. Feitelson, Jean-Pierre Prost, and Sandra Johnson Baylor. Parallel access to files in the Vesta file system. In Proceedings of Supercomputing ’93, pages 472–481, 1993.Google Scholar
  9. [8]
    Parasoft Corporation. Express version 1.0: A communication environment for parallel computers, 1988.Google Scholar
  10. [9]
    Erik DeBenedictis and Juan Miguel del Rosario. nCUBE parallel I/O software. In Proceedings of the Eleventh Annual IEEE International Phoenix Conference on Computers and Communications, pages 0117–0124, April 1992.CrossRefGoogle Scholar
  11. [10a]
    Juan Miguel del Rosario, Rajesh Bordawekar, and Alok Choudhary. Improved parallel I/O via a two-phase run-time access strategy. In IPPS ’93 Workshop on Input/Output in Parallel Computer Systems, pages 56–70, 1993.Google Scholar
  12. [10b]
    Juan Miguel del Rosario, Rajesh Bordawekar, and Alok Choudhary. Also published in Computer Architecture News 21(5), December 1993, pages 31–38.CrossRefGoogle Scholar
  13. [11]
    Juan Miguel del Rosario, Michael Harry, and Alok Choudhary. The design of VIP-FS: A virtual, parallel file system for high performance parallel and distributed computing. Technical Report SCCS-628, NPAC, Syracuse, NY 13244, May 1994.Google Scholar
  14. [12]
    Intel Supercomputer Systems Division. iPSC/2 and iPSC/860 user’s guide, April 1991. Order number: 311532–007.Google Scholar
  15. [13]
    Intel Supercomputer Systems Division. Intel Paragon XP/S user’s guide, April 1993. Order number: 312489–01.Google Scholar
  16. [14]
    Rüdiger Esser and Renate Knecht. Intel Paragon XP/S — architecture and software environment. Technical Report KFA-ZAM-IB-9305, Central Institute for Applied Mathematics, Research Center Jülich, Germany, r. esser@kfa-juelich. de, April 26 1993.Google Scholar
  17. [15]
    Dror G. Feitelson, Peter F. Corbett, Yarson Hsu, and Jean-Pierre Prost. Parallel I/O systems and interfaces for parallel computers. In C.-L. Wu, editor, Multiprocessor SystemsDesign and Integration. World Scientific, 1995. To appear.Google Scholar
  18. [16]
    Dror G. Feitelson, Peter F. Corbett, and Jean-Pierre Prost. Performance of the Vesta parallel file system. In Proceedings of the Ninth International Parallel Processing Symposium, pages 150–158, April 1995.CrossRefGoogle Scholar
  19. [17]
    N. Galbreath, W. Gropp, and D. Levine. Applications-driven parallel I/O. In Proceedings of Supercomputing ’93, pages 462–471, 1993.Google Scholar
  20. [18]
    Jay Huber, Christopher L. Elford, Daniel A. Reed, Andrew A. Chien, and David S. Blumenthal. PPFS: A high performance portable parallel file system. In Proceedings of the 9th ACM International Conference on Supercomputing,pages 385–394, Barcelona, July 1995.CrossRefGoogle Scholar
  21. [19]
    David Kotz. Disk-directed I/O for MIMD multiprocessors. In Proceedings of the 1994 Symposium on Operating Systems Design and Implementation, pages 61–74, November 1994. Updated as Dartmouth TR PCS-TR94–226 on November 8, 1994.Google Scholar
  22. [20]
    David Kotz and Nils Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Proceedings of Supercomputing ’94, pages 640–649, November 1994.CrossRefGoogle Scholar
  23. [21]
    Susan J. LoVerso, Marshall Isman, Andy Nanopoulos, William Nesheim, Ewan D. Milne, and Richard Wheeler. sfs: A parallel file system for the CM-5. In Proceedings of the 1993 Summer USENIX Conference, pages 291–305, 1993.Google Scholar
  24. [22]
    Message Passing Interface Forum. MPI: A Message-Passing Interface Standard. University of Tennessee, May 1994.Google Scholar
  25. [23]
    Steven A. Moyer and V. S. Sunderam. A parallel I/O system for high-performance distributed computing. In Proceedings of the IFIP WG10.3 Working Conference on Programming Environments for Massively Parallel Distributed Systems, 1994.Google Scholar
  26. [24]
    Steven A. Moyer and V. S. Sunderam. PIOUS: a scalable parallel I/O system for distributed computing environments. In Proceedings of the Scalable High-Performance Computing Conference, pages 71–78, 1994.CrossRefGoogle Scholar
  27. [25]
    Nils Nieuwejaar and David Kotz. A multiprocessor extension to the conventional file system interface. Technical Report PCS-TR94–230, Dept. of Computer Science, Dartmouth College, September 1994.Google Scholar
  28. [26]
    Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92–020, NAS Systems Division, NASA Ames, December 1992.Google Scholar
  29. [27]
    William J. Nitzberg. Collective Parallel I/O. PhD thesis, Department of Computer and Information Science, University of Oregon, December 1995.Google Scholar
  30. [28]
    Paul Pierce. A concurrent file system for a highly parallel mass storage system. In Proceedings of the Fourth Conference on Hypercube Concurrent Computers and Applications, pages 155–160. Golden Gate Enterprises, Los Altos, CA, March 1989.Google Scholar
  31. [29]
    Brad Rullman and David Payne. An efficient file I/O interface for parallel applications. DRAFT presented at the Workshop on Scalable I/O, Frontiers ’95, February 1995.Google Scholar
  32. [30]
    K. E. Seamons and M. Winslett. An efficient abstract interface for multidimensional array I/O. In Proceedings of Supercomputing ’94, pages 650–659, November 1994.CrossRefGoogle Scholar
  33. [31]
    K. E. Seamons and M. Winslett. Physical schemas for large multidimensional arrays in scientific computing applications. In Proceedings of the 7th International Working Conference on Scientific and Statistical Database Management, pages 218–227, September 1994.CrossRefGoogle Scholar
  34. [32]
    K. E. Seamons and M. Winslett. A data management approach for handling large compressed arrays in high performance computing. In Proceedings of the Fifth Symposium on the Frontiers of Massively Parallel Computation, pages 119–128, February 1995.Google Scholar
  35. 33]
    [ Rajeev Thakur. Runtime Support for In-Core and Out-of-Core Data-Parallel Programs. PhD thesis, Department of Electrical and Computer Engineering, Syracuse University, May 1995.Google Scholar
  36. [34]
    Rajeev Thakur, Rajesh Bordawekar, Alok Choudhary, Ravi Ponnusamy, and Tarvinder Singh. PASSION runtime library for parallel I/O. In Proceedings of the Scalable Parallel Libraries Conference, pages 119–128, October 1994.Google Scholar

Copyright information

© Kluwer Academic Publishers 1996

Authors and Affiliations

  • Peter Corbett
    • 1
  • Dror Feitelson
    • 1
  • Sam Fineberg
    • 2
  • Yarsun Hsu
    • 1
  • Bill Nitzberg
    • 2
  • Jean-Pierre Prost
    • 1
  • Marc Snirt
    • 1
  • Bernard Traversat
    • 2
  • Parkson Wong
    • 2
  1. 1.IBM T.J. Watson Research CenterYorktown HeightsUSA
  2. 2.MRJ, Inc.NASA Ames Research CenterMoffett FieldCaliforniaUSA

Personalised recommendations