Summary
This chapter centers mainly on successful programming models that map algorithms and simulations to computational resources used in high-performance computing. These resources range from group-based or departmental clusters to high-end resources available at the handful of supercomputer centers around the world. Also covered are newer programming models that may change the way we program high-performance parallel computers.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Alphaserver SC user guide, 2000. Bristol, Quadrics Supercomputer World Ltd.
R. Armstrong, D. Gannon, A. Geist, K. Keahey, S. R. Kohn, L. McInnes, S. R. Parker, and B. A. Smolinski. Toward a common component architecture for high-performance scientific computing. In Proceedings of the 8th High Performance Distributed Computing (HPDC’99), 1999. URL: http://www.cca-forum.org.
S. Balay, W. D. Gropp, L. C. McInnes, and B. F. Smith. PETSc users manual. Technical Report ANL-95/11-Revision 2.1.0, Argonne National Laboratory, 2001.
R. Bariuso and A. Knies. SHMEM’s User’s Guide. SN-2515 Rev. 2.2, Cray Research, Inc., Eagan, MN, USA, 1994.
M. Bull. OpenMP 2.5 and 3.0. In Proceedings of the Workshop on OpenMP Applications and Tools, WOMPAT 2004, Houston, TX, May 17–18 2004. (Invited talk).
P. M. Burton, B. Carruthers, G. S. Fischer, B. H. Johnson, and R.W. Numrich. Converting the halo-update subroutine in the MET Office unified model to Co-Array Fortran. In W. Zwieflhofer and N. Kreitz, editors, Developments in Teracomputing: Proceedings of the Ninth ECMWF Workshop on the Use of High Performance Computing in Meteorology, pp. 177–188. World Scientific Publishing, 2001.
J. B. Carter, J. K. Bennett, and W. Zwaenepoel. Implementation and performance of Munin. In Proceedings of the 13th ACM Symp. on Operating Systems Principles (SOSP-13), pages 152–164, 1991.
Center for Programming Models for Scalable Parallel Computing. URL: http://www.pmodels.org.
R. Chandra, L. Dagum, D. Kohr, D. Maydan, J. McDonald, and R. Menon. Parallel Programming in OpenMP. Morgan Kaufmann Publishers, San Francisco, CA, 2001.
D. Chen, S. Dwarkadas, S. Parthasarathy, E. Pinheiro, and M. L. Scott. Interweave: A middleware system for distributed shared state. In Languages, Compilers, and Run-Time Systems for Scalable Computers, pages 207–220, 2000.
E. Chow, A. Cleary, and R. Falgout. HYPRE User’s manual, version 1.6.0. Technical Report UCRL-MA-137155, Lawrence Livermore National Laboratory, Livermore, CA, 1998.
D. Clark. OpenMP: A parallel standard for the masses. IEEE Concurrency, 6(1):10–12, January–March 1998.
C. Coarfa, Y. Dotsenko, J. L. Eckhardt, and J. Mellor-Crummey. Co-array Fortran performance and potential: An NPB experimental study. In The 16th International Workshop on Languages and Compilers for Parallel Computing (LCPC 2003), College Station, Texas, October 2003.
Cray Research. Application Programmer’s Library Reference Manual, 2nd edition, Nov. 1995. Publication SR-2165.
L. Dagum and R. Menon. OpenMP: An industry standard API for shared-memory programming. IEEE Computational Science & Engineering, 5(1):46–55, January–March 1998.
S. Dong and G. E. Karniadakis. Dual-level parallelism for deterministic and stochastic CFD problems. In Proceedings of Supercomputing, SC02, Baltimore, MD, 2002.
J. Dongarra, I. Foster, G. Fox, W. D. Gropp, K. Kennedy, L. Torczon, and A. White, editors. Sourcebook of Parallel Computing. Morgan Kaufmann, 2003.
P. F. Dubois. Ten Good Practices In Scientific Programming. Computing in Science & Engineering, 1(1), January–February 1999.
S. Dwarkadas, N. Hardavellas, L. Kontothanassis, R. Nikhil, and R. Stets. Cashmere-VLM: Remote memory paging for software distributed shared memory. In Proceedings of the 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing, pages 153–159. IEEE Computer Society, Apr. 1999.
Earth Simulator home page, http://www.es.jamstec.go.jp.
T. A. El-Ghazawi, W. W. Carlson, and J. M. Draper. UPC Language Specifications Version 1.1.1, October 2003. URL: http://www.gwu.edu/~upc/docs/upc_spec_1.1.1.pdf.
T. Elrad, R. E. Filman, and A. Bader. Aspect-Oriented Programming. Communications of the ACM, 44(10):29–32, October 2001.
R. D. Falgout, J. E. Jones, and U. M. Yang. The design and implementation of hypre, a library of parallel high performance preconditioners. In A. M. Bruaset and A. Tveito, editors, Numerical Solution of Partial Differential Equations on Parallel Computers, volume 51 of Lecture Notes in Computational Science and Engineering, pages 267–294. Springer-Verlag, 2005.
M. Folk, A. Cheng, and K. Yates. HDF5: A file format and I/O library for high performance computing applications. In Proceedings of Supercomputing’99 (CD-ROM). ACM SIGARCH and IEEE, Nov. 1999.
FORTRAN 77 Binding of X3H5 Model for Parallel Programming Constructs. Draft Version, ANSI X3H5, 1992.
P. C. Forum. PCF Parallel FORTRAN Extensions. FORTRAN Forum, 10(3), September 1991. (Special issue).
Global Array Project. URL: http://www.emsl.pnl.gov/docs/global.
W. D. Gropp. Learning from the success of MPI. In B. Monien, V. K. Prasanna, and S. Vajapeyam, editors, High Performance Computing — HiPC 2001, number 2228 in Lecture Notes in Computer Science, pages 81–92. Springer, Dec. 2001.
W. D. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, B. Nitzberg, W. Saphir, and M. Snir. MPI—The Complete Reference: Volume 2, The MPI-2 Extensions. MIT Press, Cambridge, MA, 1998.
W. D. Gropp, E. Lusk, and A. Skjellum. Using MPI: Portable Parallel Programming with the Message Passing Interface, 2nd edition. MIT Press, Cambridge, MA, 1999.
W. D. Gropp, E. Lusk, and T. Sterling, editors. Beowulf Cluster Computing with Linux. MIT Press, 2nd edition, 2003.
W. D. Gropp, E. Lusk, and R. Thakur. Using MPI-2: Advanced Features of the Message-Passing Interface. MIT Press, Cambridge, MA, 1999.
R. Hempel and D. W. Walker. The emergence of the MPI message passing standard for parallel computing. Computer Standards and Interfaces, 21(1):51–62, 1999.
High Performance Fortran Forum. High Performance Fortran language specification. Scientific Programming, 2(1–2):1–170, 1993.
J. M. D. Hill, B. McColl, D. C. Stefanescu, M.W. Goudreau, K. Lang, S. B. Rao, T. Suel, T. Tsantilas, and R. H. Bisseling. BSPlib: The BSP programming library. Parallel Computing, 24(14):1947–1980, Dec. 1998.
C. A. R. Hoare. Communicating sequential processes. Communications of the ACM, 21(8):666–677, Aug. 1978.
J. Hoeflinger. Towards industry adoption of OpenMP. In Proceedings of the Workshop on OpenMP Applications and Tools, WOMPAT 2004, Houston, TX, May 17–18 2004. Invited Talk.
F. Hoffman. Writing hybrid MPI/OpenMP code. Linux Magazine, 6(4):44–48, April 2004. URL: http://www.linux-mag.com/2004-04/extreme_01.html.
Y. Hu, H. Lu, A. L. Cox, and W. Zwaenepoel. OpenMP for networks of SMPs. In Proceedings of the 13th International Parallel Processing Symposium, April 1999.
P. Hyde. Java Thread Programming. SAMS, 1999.
IEEE Standard for Information Technology-Portable Operating System Interface (POSIX). IEEE Standard No.: 1003.1, 2004.
W. Jiang, J. Liu, H.-W. Jin, D. K. Panda, W. D. Gropp, and R. Thakur. High performance MPI-2 one-sided communication over InfiniBand. Technical Report ANL/MCS-P1119-0104, Mathematics and Computer Science Division, Argonne National Laboratory, 2004.
G. Jost, J. Labarta, and J. Gimenez. What multilevel parallel programs do when you are not watching: A performance analysis case study comparing MPI/OpenMP, MLP, and nested OpenMP. In Proceedings of the Workshop on OpenMP Applications and Tools, WOMPAT 2004, pages 29–40, Houston, TX, May 17–18 2004. (Invited talk).
P. Keleher, A. L. Cox, S. Dwarkadas, and W. Zwaenepoel. TreadMarks: Distributed shared memory on standard workstations and operating systems. In Proceedings of the Winter 94 Usenix Conference, pages 115–131, January 1994.
R. A. Kendall, E. Aprà, D. E. Bernholdt, E. J. Bylaska, M. Dupuis, G. I. Fann, R. J. Harrison, J. Ju, J. A. Nichols, J. Nieplocha, T. P. Straatsma, T. L. Windus, and A. T. Wong. High performance computational chemistry; an overview of NWChem a distributed parallel application. Computer Physics Communications, 128:260–283, 2002.
M. G. Knepley, R. F. Katz, and B. Smith. Developing a geodynamics simulator with petsc. In A. M. Bruaset and A. Tveito, editors, Numerical Solution of Partial Differential Equations on Parallel Computers, volume 51 of Lecture Notes in Computational Science and Engineering, pages 413–438. Springer-Verlag, 2005.
C. Koelbel, D. B. Loveman, R. S. Schreiber, G. L. Steele, and M. E. Zosel. The High Performance Fortran Handbook. MIT Press, 1994.
B. Leasure, editor. PCF Fortran: Language Definitons, Version 3.1. The Parallel Computing Forum, Champaign, IL, 1990.
J. Li, W. Liao, A. Choudhary, R. Ross, R. Thakur, W. D. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale. Parallel netCDF: A high-performance scientific I/O interface. In Proceedings of SC2003, Nov. 2003.
Z. Li, Y. Saad, and M. Sosonkina. pARMS: A parallel version of the algebraic recursive multilevel solver. Numerical Linear Algebra with Applications, 10:485–509, 2003.
R. K. Lie Huang, Barbara Chapman. OpenMP on distributed memory via global arrays. In Proceedings of Parallel Computing 2003 (ParCo2003), Dresden, Germany, September 2–5 2003.
Message Passing Interface Forum. MPI: A Message-Passing Interface standard. International Journal of Supercomputer Applications, 8(3/4):165–414, 1994.
Message Passing Interface Forum. MPI2: A Message Passing Interface standard. International Journal of High Performance Computing Applications, 12(1–2):1–299, 1998.
Message Passing Toolkit: MPI programmer’s manual, document number: 007-3687-010, 2003. Mountain View, CA, Silicon Graphics Inc.
Mpi papers. URL: http://www.mcs.anl.gov/mpi/papers.
K. Nakajima and H. Okuda. Parallel Iterative Solvers for Unstructured Grids Using and OpenMP/MPI Hybrid Programming Model for GeoFEM Platfrom on SMP Cluster Architectures. Lecture Notes in Computer Science, 2327:437–448, 2002.
B. Nichols, D. Buttlar, and J. P. Farrel. Pthreads Programming. O’Reilly & Associates, Inc, 1996.
J. Nieplocha, R. Harrison, M. Krishnan, B. Palmer, and V. Tipparaju. Combining shared and distributed memory models: Evolution and recent advancements of the Global Array Toolkit. In Proceedings of POOHL’2002 workshop of ICS-2002, New York, NY, 2002.
J. Nieplocha, R. J. Harrison, and R. J. Littlefield. Global Arrays: A portable “shared memory” programming model for distributed memory computers. In Proceedings of Supercomputing 1994, SC94, pages 340–349, 1994.
J. Nieplocha, R. J. Harrison, and R. J. Littlefield. Global Arrays: A nonuniform memory access programming model for high-performance computers. The Journal of Supercomputing, 10:197–220, 1996.
R. W. Numrich, J. Reid, and K. Kim. Writing a multigrid solver using Co-Array Fortran. In B. Kågströom, J. Dongarra, E. Elmroth, and J. Waśsniewski, editors, Applied Parallel Computing: Large Scale Scientific and Industrial Problems, volume 1541 of Lecture Notes in Computer Science, pages 390–399. Springer, 1998.
R. W. Numrich and J. K. Reid. Co-Array Fortran for parallel programming. ACM Fortran Forum, 17(2):1–31, 1998.
OpenMP Architecture Review Board. OpenMP Fortran Application Program Interface, Version 2.0. November 2000. URL: http://www.openmp.org/drupal/mp-documents/fspec20.pdf.
OpenMP Architecture Review Board. OpenMP C and C++ Application Program Interface, Version 2.0. March 2002. URL: http://www.openmp.org/drupal/mp-documents/cspec20.pdf.
OpenMP Architecture Review Board home page, http://www.openmp.org.
K. Parzyszek and R. A. Kendall. GPSHMEM: Application to kernel benchmarks. In Proceedings of the Fourteenth IASTED International Conference on Parallel and Distributed Computing and Systems (PDCS 2002), pages 404–409. ACTA Press, Anaheim, CA, 2002.
K. Parzyszek, J. Nieplocha, and R. A. Kendall. A generalized portable SHMEM library for high performance computing. In M. Guizani and X. Shen, editors, Proceedings of the IASTED Parallel and Distributed Computing and Systems 2000, pages 401–406. IASTED, Calgary, 2000.
Y. Saad. SPARSKIT: A basic tool kit for sparse matrix computations. Technical Report 90-20, NASA Ames Research Center, Moffett Field, CA, 1990.
H. Sakagami, H. Murai, Y. Seo, and M. Yokokawa. 14.9 TFLOPS three-dimensional fluid simulation for fusion science with HPF on the Earth Simulator. In Proceedings of Supercomputing, 2002.
Scali Library User’s Guide, 2002. Published by Scali, Oslo, Norway.
C. L. Seitz. The cosmic cube. Communications of the ACM, 28(1):22–33, Jan. 1985.
B. Smith, P. Bjørstad, and W. D. Gropp. Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations. Cambridge University Press, New York, 1996.
M. Snir, S. W. Otto, S. Huss-Lederman, D. W. Walker, and J. Dongarra. MPI: The Complete Reference. MIT Press, Cambridge, MA, 1995.
T. Straatsma, E. Aprà, T. Windus, W. E. d. J. E. J. Bylaska, S. Hirata, M. Valiev, M. T. Hackler, L. L. Pollack, R. J. Harrison, M. Dupuis, D. Smith, J. Nieplocha, V. Tipparaju, M. Krishnan, A. A. Auer, E. Brown, G. Cisneros, G. I. Fann, H. Fruchtl, J. Garza, K. Hirao, R. A. Kendall, J. Nichols, K. Tsemekhman, K. Wolinski, J. Anchell, D. Bernholdt, P. Borowski, T. Clark, D. Clerc, H. Dachsel, M. Deegan, K. K. Dyall, D. Elwood, E. Glendening, M. Gutowski, A. Hess, J. Jaffe, B. Johnson, J. Ju, R. Kobayashi, R. Kutteh, Z. Lin, R. Littlefield, X. Long, B. Meng, T. Nakajima, S. Niu, M. Rosing, G. Sandrone, M. Stave, H. Taylor, G. Thomas, J. van Lenthe, A. Wong, and Z. Zhang. NWChem, A computational chemistry package for parallel computers, Version 4.6, 2004. Pacific Northwest National Laboratory, Richland, WA.
R. Thakur, W. D. Gropp, and B. Toonen. Minimizing synchronization overhead in the implementation of MPI one-sided communication. In D. Kranzlmüller, P. Kacsuk, and J. Dongarra, editors, Recent Advances in Parallel Virtual Machine and Message Passing Interface, Lecture Notes in Computer Science, pages 57–67. Springer Verlag, 2004. 11th European PVM/MPI User’s Group Meeting, Budapest, Hungary.
The Cluster Enabled Omni OpenMP Compiler. URL: http://phase.hpcc.jp/Omni/Omni-doc/omni-scash.html.
The Open Group. System Interfaces and Headers, Issue 4, Version 2. 1992. URL: http://www.opengroup.org/public/pubs/catalog/c435.htm.
K. Yelick, L. Semenzato, G. Pike, C. Miyamoto, B. Liblit, A. Krishnamurthy, P. Hilfinger, S. Graham, D. Gay, P. Colella, and A. Aiken. Titanium: A high-performance Java dialect. Concurrency: Practice And Experience, 10(11–13):825–836, 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kendall, R.A., Sosonkina, M., Gropp, W.D., Numrich, R.W., Sterling, T. (2006). Parallel Programming Models Applicable to Cluster Computing and Beyond. In: Bruaset, A.M., Tveito, A. (eds) Numerical Solution of Partial Differential Equations on Parallel Computers. Lecture Notes in Computational Science and Engineering, vol 51. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-31619-1_1
Download citation
DOI: https://doi.org/10.1007/3-540-31619-1_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29076-6
Online ISBN: 978-3-540-31619-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)