Chapter

Recent Advances in Parallel Virtual Machine and Message Passing Interface

Volume 3666 of the series Lecture Notes in Computer Science pp 67-75

Scalable Fault Tolerant MPI: Extending the Recovery Algorithm

  • Graham E. FaggAffiliated withDept. of Computer Science, The University of Tennessee
  • , Thara AngskunAffiliated withDept. of Computer Science, The University of Tennessee
  • , George BosilcaAffiliated withDept. of Computer Science, The University of Tennessee
  • , Jelena Pjesivac-GrbovicAffiliated withDept. of Computer Science, The University of Tennessee
  • , Jack J. DongarraAffiliated withDept. of Computer Science, The University of Tennessee

* Final gross prices may vary according to local VAT.

Get Access

Abstract

Fault Tolerant MPI (FT-MPI) [6] was designed as a solution to allow applications different methods to handle process failures beyond simple check-point restart schemes. The initial implementation of FT-MPI included a robust heavy weight system state recovery algorithm that was designed to manage the membership of MPI communicators during multiple failures. The algorithm and its implementation although robust, was very conservative and this effected its scalability on both very large clusters as well as on distributed systems. This paper details the FT-MPI recovery algorithm and our initial experiments with new recovery algorithms that are aimed at being both scalable and latency tolerant. Our conclusions shows that the use of both topology aware collective communication and distributed consensus algorithms together produce the best results.