Parallel IO Support for Meta-computing Applications: MPI_Connect IO Applied to PACX-MPI

  • Graham E. Fagg
  • Edgar Gabriel
  • Michael Resch
  • Jack J. Dongarra
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2131)

Abstract

Parallel IO (PIO) support for larger scale computing is becoming more important as application developers better understand its importance in reducing overall execution time by avoiding IO overheads. This situation has been made more critical as processor speed and overall system size has increased at a far greater rate than sequential IO performance. Systems such as MPI_Connect and PACX-MPI allow multiple MPPs to be interconnected, complicating IO issues further. MPI_Connect implemented Parallel IO support for distributed applications in the MPI_Conn_IO package by transferring complete sections of files to remote machines, supporting the case that all the applications and the file storage were completely distributed. This system had a number of performance drawbacks compared to the more common usage of metacomputing where some files and applications have an affinity to a home site and thus less data transfer is required. Here we present the new PACX-MPI PIO system based initially on MPI_Connect IO, and attempt to demonstrate multiple methods of handling MPI PIO that cover a greater number of possible usage scenarios. Given are some preliminary performance results as well as a comparison to other PIO grid systems such as the Asian Pacific GridFarm, and Globus gridFTP, GASS and RIO.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    G. E. Fagg, K. S. London, and J. J. Dongarra, “MPI_Connect: managing heterogeneous MPI applications interoperation and process control”, in V. Alexandrov and J. Dongarra, editors, Recent advances in Parallel Virtual Machine and Message Passing Interface, volume 1497 of Lecture notes of Computer Science, pages 93–96. Springer, 1998. 5th European PVM/MPI User’s Group Meeting.Google Scholar
  2. 2.
    D. Cronk, G. Fagg, and S. Moore, “Parallel I/O for EQM Applications”, Department of Computer Science technical report, University of Tenneesee, Knoxville, July 2000.Google Scholar
  3. 3.
    E. Gabriel, M. Resch, T. Beisel and R. Keller“Distributed Computing in a heterogeneous computing environment”, in V. Alexandrov and J. Dongarra, editors, Recent advances in Parallel Virtual Machine and Message Passing Interface, volume 1497 of Lecture notes of Computer Science, pages 180–188. Springer, 1998. 5th European PVM/MPI User’s Group Meeting.Google Scholar
  4. 4.
    T. Imamura, Y. Tsujita, H. Koide, and H. Takemiya, “An architecture of Stampi: MPI library on a cluster of parallel computers”, in J. Dongarra, P. Kacsuk, and N. Podhorszki, editors, Recent advances in Parallel Virtual Machine and Message Passing Interface, volume 1908 of Lecture notes of Computer Science, pages 200–207. Springer, 2000. 7th European PVM/MPI User’s Group Meeting.Google Scholar
  5. 5.
    Rolf Rabenseifner and Alice E. Koniges,’ Effective File-I/O Bandwidth Benchmark’, in A. Bode, T. Ludwig, R. Wissmüller, editors, Proceedings of Euro-Par 200, pages 1273–1283, Springer 2000.Google Scholar
  6. 6.
    B. Allock et all, “Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing”, submitted to IEEE Mass Storage Conference, April 2001.Google Scholar
  7. 7.
    R. Thakur, W. Gropp, and E. Lusk, “On Implementing MPI-IO Portably and with High Performance”, in Proc. Of the Sixth Workshop on I/O in Parallel and Distributed Systems, May 1999, pages 23–32.Google Scholar
  8. 8.
    J. Bester, I. Foster, C. Kesselmann, J. Tedesco, S. Tuecke, “GASS: A Data Movement and Access Service for Wide Area Computing Systems”, in Sixth Workshop on I/O in Parallel and Distributed Systems, 1999.Google Scholar
  9. 9.
    I. Foster, D. Kohr, R. Krishnaiyer, J. Mogill, “Remote I/O: Fast Access to Distant Storage”, in Proc. Workshop on I/O in Parallel and Distributed Systems, (IOPADS), pages 14–25, 1997.Google Scholar
  10. 10.
    O. Tatebe et all., “Grid Data Fram for Petascale Data Instensive Computing”, Elechtro-technical Laboratories, Technical Report, TR-2001-4, http://datafarm.apgrid.org.
  11. 11.
    B. Tierny, W. Johnston, J. Lee, M. Thompson, “A Data Intensive Distributed Computing Architecture for Grid Applications”, Future Generation Computer Systems, volume 16 no 5, pages 473–481, Elsevier Science, 2000.CrossRefGoogle Scholar
  12. 12.
    J. Morris, M. Satyanarayanan, M. Conner, J. Howard, D. Rosenthal and F. Smith. “Andrew: A distributed personal computing environment”, Communications of the ACM, 29(3):184–201, 1986.CrossRefGoogle Scholar
  13. 13.
    William Gropp, et. Al., MPI-The Complete Reference, Volume 2, The MPI Extensions, The MIT Press, Cambridge, MA 1999Google Scholar
  14. 14.
    William Gropp, Ewing Lusk, and Rajeev Thakur, Using MPI-2, Advanced Features of the Message-Passing Interface, The MIT Press, Cambridge, MA 1999CrossRefGoogle Scholar
  15. 15.
    James S. Plank, Micah Beck, Wael R. Elwasif, Terry Moore, Martin Swany, Rich Wolski “The Internet Backplane Protocol: Storage in the Network”, NetStore99: The Network Storage Symposium, (Seattle, WA, 1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Graham E. Fagg
    • 1
  • Edgar Gabriel
    • 2
  • Michael Resch
    • 2
  • Jack J. Dongarra
    • 1
  1. 1.Department of Computer ScienceUniversity of TennesseeKnoxvilleUSA
  2. 2.Performance Computing Center StuttgartStuttgartGermany

Personalised recommendations