Skip to main content
SpringerLink
Log in

MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared memory

  • Published:
Computing Aims and scope Submit manuscript

Abstract

Hybrid parallel programming with the message passing interface (MPI) for internode communication in conjunction with a shared-memory programming model to manage intranode parallelism has become a dominant approach to scalable parallel programming. While this model provides a great deal of flexibility and performance potential, it saddles programmers with the complexity of utilizing two parallel programming systems in the same application. We introduce an MPI-integrated shared-memory programming model that is incorporated into MPI through a small extension to the one-sided communication interface. We discuss the integration of this interface with the MPI 3.0 one-sided semantics and describe solutions for providing portable and efficient data sharing, atomic operations, and memory consistency. We describe an implementation of the new interface in the MPICH2 and Open MPI implementations and demonstrate an average performance improvement of 40 % to the communication component of a five-point stencil solver.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. MPI Forum (2012) MPI: a message-passing interface standard. version 3.0

  2. Smith L, Bull M (2001) Development of mixed mode MPI/OpenMP applications. Sci Program 9(2,3):83–98

    Google Scholar 

  3. Rabenseifner R, Hager G, Jost G (2009) Hybrid MPI/OpenMP parallel programming on clusters of multi-core SMP nodes. In: Proceedings of the 17th Euromicro international conference on parallel, distributed and network-based processing

  4. Demaine E (1997) A threads-only MPI implementation for the development of parallel programs. In: Proceedings of the 11th international symposium on HPC systems. pp 153–163

  5. Bhargava P (1997) MPI-LITE: multithreading support for MPI. http://pcl.cs.ucla.edu/projects/sesame/mpi_lite/mpi_lite.html

  6. Shen K, Tang H, Yang T (1999) Adaptive two-level thread management for fast MPI execution on shared memory machines. In: Proceedings of the ACM/IEEE conference on supercomputing

  7. Tang H, Shen K, Yang T (2000) Program transformation and runtime support for threaded MPI execution on shared memory machines. ACM Trans Program Lang Syst 22:673–700

    Article  Google Scholar 

  8. Pérachec M, Carribault P, Jourdren H (2009) MPC-MPI: an MPI implementation reducing the overall memory consumption. In: Proceedings of EuroPVM/MPI 2009, Springer, pp 94–103

  9. Shirley D (2000) Enhancing MPI applications through selective use of shared memory on SMPs. In: Proceedings of the 1st SIAM conference on CSE

  10. Los Alamos National Laboratory (2001) Unified parallel software users’ guide and reference manual. http://public.lanl.gov/ups/Doc_Directory/UserGuide/UserGuide.pdf

  11. Hoefler T, Dinan J, Buntinas D, Balaji P, Barrett B, Brightwell R, Gropp W, Kale V, Thakur R (2012) Leveraging MPIs one-sided communication interface for shared-memory programming. In: Träff J, Benkner S, Dongarra J (eds) Recent advances in the message passing interface. vol 7490, pp 132–141

  12. Taft JR (2001) Achieving 60 GFLOP/s on the production CFD code OVERFLOW-MLP. Parallel Comput 27(4):521–536

    Article  MATH  Google Scholar 

  13. Boehm HJ (2005) Threads cannot be implemented as a library. In: Proceedings of the 2005 ACM SIGPLAN conference on programming language design and implementation. PLDI ’05, New York, NY, USA, ACM pp 261–268

  14. Boehm HJ, Adve SV (2012) You do not know jack about shared variables or memory models. Commun. ACM 55(2):48–54

    Article  Google Scholar 

  15. Aho AV, Sethi R, Ullman JD (1986) Compilers: principles, techniques, and tools. Addison-Wesley Longman Publishing Co. Inc., Boston

    Google Scholar 

  16. Manson J, Pugh W, Adve SV (2005) The Java memory model. In: Proceedings of the 32nd ACM SIGPLAN-SIGACT symposium on principles of programming languages. POPL ’05, New York, ACM pp 378–391

  17. Boehm HJ, Adve SV (2008) Foundations of the C++ concurrency memory model. SIGPLAN Not 43(6):68–78

    Article  Google Scholar 

  18. Lee EA (2006) The problem with threads. Computer 39(5):33–42

    Article  Google Scholar 

  19. Heroux MA, Brightwell R, Wolf MM (2011) Bi-modal MPI and MPI+threads computing on scalable multicore systems. IJHPCA (Submitted)

  20. Sandia National Laboratories (2012) Mantevo project. http://www.mantevo.org

  21. Saad Y (2003) Iterative methods for sparse linear systems. Society for Industrial and Applied Mathematics

  22. Saltz JH (1990) Aggregation methods for solving sparse triangular systems on multiprocessors. SIAM J Sci Stat Comput 11(1):123–144

    Article  MathSciNet  MATH  Google Scholar 

  23. Wolf MM, Heroux MA, Boman EG (2010) Factors impacting performance of multithreaded sparse triangular solve. Technical report SAND2010-0331 presented at VECPAR’10

  24. Esler KP, Kim J, Ceperley DM, Purwanto W, Walter EJ, Krakauer H, Zhang S, Kent PRC, Hennig RG, Umrigar C, Bajdich M, Koloren J, Mitas L, Srinivasan A (2008) Quantum monte carlo algorithms for electronic structure at the petascale; the endstation project. J Phys 125(1):012057

    Google Scholar 

  25. Wagner LK, Bajdich M, Mitas L (2009) Qwalk: a quantum monte carlo program for electronic structure. J Comput Phys 228(9):3390–3404

    Article  Google Scholar 

  26. Esler KP Einspline libaray. Online: http://einspline.svn.sourceforge.net/

  27. Niu Q, Dinan J, Tirukkovalur S, Mitas L, Wagner L, Sadayappan P (2012) A global address space approach to automated data management for parallel quantum Monte Carlo applications. In: Proceedings 19th international conference on high performance computing. HiPC’12

  28. Smith L, Kent P (2000) Development and performance of a mixed OpenMP/MPI quantum Monte Carlo code. Concurr Pract Exp 12(12):1121–1129

    Article  MATH  Google Scholar 

  29. Esler KP, Kim J, Ceperley DM, Shulenburger L (2012) Accelerating quantum Monte Carlo simulations of real materials on GPU clusters. Comput Sci Eng 14(1):40–51

    Article  Google Scholar 

Download references

Acknowledgments

We thank the members of the MPI Forum and the MPI community for their efforts in creating the MPI 3.0 specification. In addition, we thank Jeff RḢammond for reviewing a draft of this article. This work was supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research, under Contract DE-AC02-06CH11357, under award number DE-FC02-10ER26011 with program manager Sonia Sachs, under award number DE-FG02-08ER25835, and as part of the Extreme-scale Algorithms and Software Institute (EASI) by the Department of Energy, Office of Science, U.S. DOE award DE-SC0004131. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energys National Nuclear Security Administration, under contract DE-AC-94AL85000.

Author information

Authors and Affiliations

  1. ETH Zurich, Zurich, Switzerland

    Torsten Hoefler

  2. Argonne National Laboratory, Argonne, IL, USA

    James Dinan, Darius Buntinas, Pavan Balaji & Rajeev Thakur

  3. Sandia National Laboratories, Sandia, NM, USA

    Brian Barrett & Ron Brightwell

  4. University of Illinois at Urbana-Champaign, Urbana, IL, USA

    William Gropp & Vivek Kale

Authors
  1. Torsten Hoefler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James Dinan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Darius Buntinas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pavan Balaji
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Brian Barrett
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ron Brightwell
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. William Gropp
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Vivek Kale
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Rajeev Thakur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Torsten Hoefler.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hoefler, T., Dinan, J., Buntinas, D. et al. MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared memory. Computing 95, 1121–1136 (2013). https://doi.org/10.1007/s00607-013-0324-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-013-0324-2

Keywords

Mathematics Subject Classification

Search

Navigation