The Journal of Supercomputing

, Volume 11, Issue 2, pp 119–136 | Cite as

Shared Memory Programming in Metacomputing Environments: The Global Array Approach

  • Jarek Nieplocha
  • Robert Harrison


The performance of the Global Array shared-memory nonuniform memory-access programming model is explored in a wide-area-network (WAN) distributed supercomputer environment. The Global Array model is extended by introducing a concept of mirrored arrays that thanks to the caching and user-controlled consistency of the shared data structure scan reduce the application sensitivity to the network latency. Latencies and bandwidths for remote memory access are studied, and the performance of a large application from computational chemistry is evaluated using both fully distributed and also mirrored arrays. Excellent performance can be obtained with mirroring if even modest (0.5 MB/s) network bandwidth is available.

Metacomputing shared-memory programming NUMA memory architecture global arrays distributed arrays 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    C. Amza, A. L. Cox, S. Dwarkadas, P. Kohler, H. Lu, R. Rajamony, W. Yu, and W. Zwaenepoel. “Tread-Marks: Shared memory computing on networks of workstations”, IEEE Computer, 29(2):18–28, 1996.Google Scholar
  2. 2.
    Anonymous Ftp:, 1997.Google Scholar
  3. 3.
    D. E. Bernholdt, E. Aprà, H. A. Früchtl, M. F. Guest, R. J. Harrison, R. A. Kendall, R. A. Kutteh, X. Long, J. B. Nicholas, J. A. Nichols, H. L. Taylor, A. T. Wong, G. I. Fann, R. J. Littlefield, and J. Nieplocha. ”Parallel Computational Chemistry Made Easier: The Development of NWChem,” Int. J. Quantum Chem. Symp., 29:475–483, 1995.Google Scholar
  4. 4.
    N. Carriero and D. Gelernter. “Linda in Context”, Communications of the ACM, 32(4):444–458, 1989.Google Scholar
  5. 5.
    T. von Eicken, V. Avula, A. Basu, and V. Buch. “Low–latency communication over ATM networks using active messages”. IEEE Micro, 15(1):46–53, 1995.Google Scholar
  6. 6.
    T. De Fanti, I. Foster, M. Papka, R. Stevens, and T. Kuhfuss. Overview of the I–WAY: Wide Area Visual Supercomputing, International Journal of Supercomputer Applications, 10(2):123–130, 1996.Google Scholar
  7. 7.
    I. Foster, J. Geisler, B. Nickless, W. Smith, and S. Tuecke. “Software Infrastructure for High–Performance Distributed Computing”, In Proceedings 5–th IEEE Int. Symp. HPDC, pages 562–570, IEEE Computer Society Press, 1996.Google Scholar
  8. 8.
    A. S. Grimshaw, W. A. Wulf, J. C. French, A. C. Weaver, and P. F. Reynolds Jr. “Legion: The Next Logical Step Toward a Natiowide Virtual Computer”, Technical Report CS–94–21, Dept. Computer Science, University of Virginia, 1994.Google Scholar
  9. 9.
    R. J. Harrison. “Portable Tools and Applications for Parallel Computers,” Int. J. Quant. Chem., pp. 847–863, 40, 1991.Google Scholar
  10. 10.
    R. J. Harrison, M. F. Guest, R. A. Kendall, D. E. Bernholdt, A. T. Wong, M. S. Stave, J. L. Anchell, A. C. Hess, R. J. Littlefield, G. I. Fann, J. Nieplocha, G. S. Thomas, D. Elwood, J. Tilson, R. L. Shepard, A. F. Wagner, I. T. Foster, E. Lusk, and R. Stevens. Toward high–performance computational chemistry: II. A scalable self–consistent field program. J. Comp. Chem., 17(1):124–132, 1996.Google Scholar
  11. 11.
    C. Koelbel, D. Loveman, R. Schreiber, G. S. Jr., and M. E. Zosel. The High Performance Fortran Handbook. The MIT Press, Cambridge, MA, 1994.Google Scholar
  12. 12.
    Message Passing Interface Forum, MPI: A Message–Passing Interface, University of Tennessee, Knoxville, Tenn., May 5, 1994.Google Scholar
  13. 13.
    J. Nieplocha, R. J. Harrison, and I. Foster. Explicit Management of Memory Hierarchy, In L. Grandinetti and J. Kowalik, editor, Advances in High Performance Computing, Kluwer Academic, 1997 (in press).Google Scholar
  14. 14.
    J. Nieplocha, R. J. Harrison, and R. J. Littlefield. “Global Arrays: A Portable Shared–Memory Programming Model for Distributed–Memory Computers”, In Proc. Supercomputing'94, pages 340–349. IEEE Computer Society Press, 1994.Google Scholar
  15. 15.
    J. Nieplocha, R. J. Harrison, and R. J. Littlefield. “Global Arrays: A Nonuniform Memory Access Programming Model for High–Performance Computers”, The Journal of Supercomputing, 10:169–189, 1996.Google Scholar
  16. 16.
    S. Pakin, M. Lauria, and A. Chien. “High Performance Messaging on Workstations: Illinois Fast Messages (FM) for Myrinet”, Proc. Supercomputing'95, IEEE Computer Society, 1995.Google Scholar
  17. 17.
    M. Oguchi, H. Aida, and T. Saito. “A Proposal for a DSM Architecture Suitable for a Widely Distributed Environment and its Evaluation”, Proceedings 4–th IEEE Int. Symp. HPDC, pages 32–39, IEEE Computer Society Press, 1995.Google Scholar
  18. 18.
    J. B. Weissman and A. S. Grimshaw. “A federated model forscheduling in wide–area systems”, In Proceedings 5–th IEEE Int. Symp. HPDC, pp. 542–550, IEEE Computer Society Press, 1996.Google Scholar

Copyright information

© Kluwer Academic Publishers 1997

Authors and Affiliations

  • Jarek Nieplocha
    • 1
  • Robert Harrison
    • 1
  1. 1.Pacific Northwest National LaboratoryRichland

Personalised recommendations