Using Triggered Operations to Offload Rendezvous Messages

  • Brian W. Barrett
  • Ron Brightwell
  • K. Scott Hemmert
  • Kyle B. Wheeler
  • Keith D. Underwood
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6960)


Historically, MPI implementations have had to choose between eager messaging protocols that require buffering and rendezvous protocols that sacrifice overlap and strong independent progress in some scenarios. The typical choice is to use an eager protocol for short messages and switch to a rendezvous protocol for long messages. If overlap and progress are desired, some implementations offer the option of using a thread. We propose an approach that leverages triggered operations to implement a long message rendezvous protocol that provides strong progress guarantees. The results indicate that a triggered operation based rendezvous can achieve better overlap than a traditional rendezvous implementation and less wasted bandwidth than an eager long protocol.


Message Passing Interface Message Size Counting Event Message Protocol Large Message 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Brightwell, R.: A Comparison of Three MPI Implementations for Red Storm. In: Di Martino, B., Kranzlmüller, D., Dongarra, J. (eds.) EuroPVM/MPI 2005. LNCS, vol. 3666, pp. 425–432. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  2. 2.
    Brightwell, R., Maccabe, A.B., Riesen, R.: Design, implementation, and performance of MPI on Portals 3.0. International Journal of High Performance Computing Applications 17(1), 7–20 (2003)CrossRefzbMATHGoogle Scholar
  3. 3.
    Brightwell, R., Underwood, K.D.: Evaluation of an eager protocol optimization for MPI. In: Dongarra, J., Laforenza, D., Orlando, S. (eds.) EuroPVM/MPI 2003. LNCS, vol. 2840, pp. 327–334. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  4. 4.
    Dickman, L., Lindahl, G., Olson, D., Rubin, J., Broughton, J.: PathScale InfiniPath: A First Look. In: Proceedings of the 13th Symposium on High Performance Interconnects (HOTI 2005) (August 2005)Google Scholar
  5. 5.
    Franke, H., Wu, C.E., Riviere, M., Pattnaik, P., Snir, M.: MPI programming environment for IBM SP1/SP2. In: Proceedings of the 15th International Conference on Distributed Computing Systems (1995)Google Scholar
  6. 6.
    Gropp, W., Lusk, E., Doss, N., Skjellum, A.: A high-performance, portable implementation of the MPI message passing interface standard. Parallel Computing 22(6), 789–828 (1996)CrossRefzbMATHGoogle Scholar
  7. 7.
    Hemmert, K.S., Barrett, B., Underwood, K.D.: Using triggered operations to offload collective communication operations. In: Keller, R., Gabriel, E., Resch, M., Dongarra, J. (eds.) EuroMPI 2010. LNCS, vol. 6305, pp. 249–256. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Kumar, R., Mamidala, A.R., Koop, M.J., Santhanaraman, G., Panda, D.K.: Lock-Free Asynchronous Rendezvous Design for MPI Point-to-Point Communication. In: Lastovetsky, A., Kechadi, T., Dongarra, J. (eds.) EuroPVM/MPI 2008. LNCS, vol. 5205, pp. 185–193. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Myricom, Inc.: Myrinet Express (MX): A high performance, low-level, message-passing interface for Myrinet (July 2003),
  10. 10.
    Petrini, F., chun Feng, W., Hoisie, A., Coll, S., Frachtenberg, E.: The Quadrics Network: High-Performance Clustering Technology. IEEE Micro 22(1), 46–57 (2002)CrossRefGoogle Scholar
  11. 11.
    Rashti, M.J., Afsahi, A.: Improving Communication Progress and Overlap in MPI Rendezvous Protocol over RDMA-enabled Interconnects. In: Proceedings of the 22nd International Symposium on High Performance Computing Systems, pp. 95–101 (June 2008)Google Scholar
  12. 12.
    Rashti, M.J., Afsahi, A.: A speculative and adaptive MPI rendezvous protocol over RDMA-enabled interconnects. International Journal of Parallel Programming 37, 223–246 (2009)CrossRefzbMATHGoogle Scholar
  13. 13.
    Riesen, R.E., Pedretti, K.T., Brightwell, R., Barrett, B.W., Underwood, K.D., Hudson, T.B., Maccabe, A.B.: The Portals 4.0 message passing interface. Tech. Rep. SAND2008-2639, Sandia National Laboratories (April 2008)Google Scholar
  14. 14.
    Rodrigues, A.F., Hemmert, K.S., Barrett, B.W., Kersey, C., Oldfield, R., Weston, M., Risen, R., Cook, J., Rosenfeld, P., CooperBalls, E., Jacob, B.: The structural simulation toolkit. SIGMETRICS Perform. Eval. Rev. 38, 37–42 (2011), CrossRefGoogle Scholar
  15. 15.
    Sur, S., Jin, H.W., Chai, L., Panda, D.K.: Rdma read based rendezvous protocol for mpi over infiniband: design alternatives and benefits. In: Proceedings of the eleventh ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP 2006, pp. 32–39. ACM, New York (2006), Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Brian W. Barrett
    • 1
  • Ron Brightwell
    • 1
  • K. Scott Hemmert
    • 1
  • Kyle B. Wheeler
    • 1
  • Keith D. Underwood
    • 2
  1. 1.Sandia National LaboratriesAlbuquerque
  2. 2.Intel CorporationHillsboroUSA

Personalised recommendations