Cluster Computing

, Volume 6, Issue 2, pp 115–124 | Cite as

One-Sided Communication on Clusters with Myrinet

  • Jarek Nieplocha
  • Edoardo Apra
  • Jialin Ju
  • Vinod Tipparaju


This paper describes an efficient implementation of one-sided communication on top of the GM low-level message-passing library for clusters with Myrinet. This approach is compatible with shared memory, exploits pipelining, nonblocking communication, and overlapping memory registration with memory copy to maximize the transfer rate. The paper addresses critical design issues faced on the commodity clusters and then describes possible solutions for matching the low-level network protocol with user-level interfaces. The performance implications of the design decisions are presented and discussed in context of a standalone communication benchmark as well as two applications. Finally, the paper offers some indications on what additional features would be desirable in a communication library like GM to better support one-sided communication.

one-sided communication remote memory operations clusters Myrinet 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    S. Lumetta, A. Mainwaring and D. Culler, Multi-protocol active messages on a cluster of SMP's, in: Proceedings of Supercomputing '97 (1997).Google Scholar
  2. [2]
    H. Tezuka, A. Hori, Y. Ishikawa and M. Sato, PM: An operating system coordinated high performance communication library, in: High Performance Computing and Networking, Lecture Notes in Computer Science, Vol. 1225 (Springer, 1997).Google Scholar
  3. [3]
    L. Prylli and B. Tourancheau, BIP: A new protocol designed for high performance networking on Myrinet, in: Workshop PC-NOW, IPPS/SPDP98 (Orlando, USA, 1998).Google Scholar
  4. [4]
    S. Parkin, M. Lauria, A. Chien et al., High Performance Virtual Machines (HPVM): Clusters with supercomputing APIs and performance, in: 8th SIAM Conference on Parallel Processing for Scientific Computing (PP97) (1997).Google Scholar
  5. [5]
    Myricom, The GM Message Passing System (10/16/1999).Google Scholar
  6. [6]
    J. Nieplocha and B. Carpenter, ARMCI: A portable remote memory copy library for distributed array libraries and compiler run-time systems, in: Proc. RTSPP IPPS/SDP'99 (1999).Google Scholar
  7. [7]
    K. Parzyszek, J. Nieplocha and R. Kendall, A generalized portable SHMEM library for high performance computing, in: Proceedings Int. Conf. on Parallel and Distributed Computing and Systems, PDCS-2000 (2000).Google Scholar
  8. [8]
    R. Bariuso and A. Knies, SHMEM's User's Guide, SN-2516, rev. 2.2 (Cray Research, Inc., Eagan, MN, 1994).Google Scholar
  9. [9]
    J. Nieplocha, J. Ju and T.P. Straatsma, A multiprotocol communication support for the global address space programming model on the IBM SP, in: Proc. EuroPar-2000, Lecture Notes in Computer Science, Vol. 1900 (Springer, 2000).Google Scholar
  10. [10]
    K. Langendoen, J. Romein, R. Bhoedjang and H. Bal, Integrating polling, interrupts, and thread management, in: Proc. Frontiers'96 (1996).Google Scholar
  11. [11]
    J. Hsieh, T. Leng, V. Mashayekhi and R. Rooholamini, Architectural and performance evaluation of GigaNet and Myrinet interconnects on clusters of small-scale SMP servers, in: Proceedings SC2000, Scholar
  12. [12]
    M. Banikazemi, R. Govindaraju, R. Blackmore and D.K. Panda, Implementing efficient MPI on LAPI for the IBM RS/600 SP systems: Experiences and performance evaluation, in: Proc. IPPS99 (1999).Google Scholar
  13. [13]
    A. Gallatin, J. Chase and K. Yocum, Trapeze/IP: TCP/IP at near-gigabit speeds, in: 1999 USENIX Technical Conference, June 1999.Google Scholar
  14. [14]
    K. Moldeklev and P. Gunningberg, How a large ATM MTU causes deadlocks in TCP data transfers, IEEE/ACMTransactions on Networking 3(4) (August 1995).Google Scholar
  15. [15]
    J. Nieplocha, J. Ju and E. Apra, One-sided communication on SMP clusters with Myrinet using the GM message-passing library, in: Proc. CAC'01/IPDPS'01 (2001).Google Scholar
  16. [16]
    Myricom, Portable MPI Model Implementation over GM, version 1.2.1 (file README-GM) (23 June, 2000).Google Scholar
  17. [17]
    L.A. Giannini and A. Chien, A software architecture for Global address space on communication on clusters: Put/Get on Fast Messages, in: 7th Int. IEEE Symposium on High Performance Distributed Computing, HPDC-7 (1998).Google Scholar
  18. [18]
    S.C. Woo, M.O. Ohara, E. Torrie, J.P. Singh and A. Gupta, The SPLASH-2 programs: Characterization and methodological considerations, in: Proc. 22nd International Symposium on Computer Architecture (1995).Google Scholar
  19. [19]
    High Performance Computational Chemistry Group, NWChem, A Computational Chemistry Package for Parallel Computers, Version 4.0, Pacific Northwest National Laboratory, Richland, WA (2000), Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • Jarek Nieplocha
    • 1
  • Edoardo Apra
    • 1
  • Jialin Ju
    • 2
  • Vinod Tipparaju
    • 3
  1. 1.Pacific Northwest National LaboratoryUSA
  2. 2.Tivoli Systems IncUSA
  3. 3.Ohio State UniversityUSA

Personalised recommendations