Skip to main content

Introducing Cray OpenSHMEMX - A Modular Multi-communication Layer OpenSHMEM Implementation

  • Conference paper
  • First Online:
OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Extreme Heterogeneity (OpenSHMEM 2018)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11283))

Included in the following conference series:


SHMEM has a long history as a parallel programming model. It is extensively used since 1993, starting from Cray T3D systems. For the past two decades SHMEM library implementation in Cray systems evolved through different generations. The current generation of the SHMEM implementation for Cray XC and XK systems is called Cray SHMEM. It is a proprietary SHMEM implementation from Cray Inc. In this work, we provide an in-depth analysis of need for a new SHMEM implementation and then introduce the next evolution of Cray SHMEM implementation for current and future generation Cray systems. We call this new implementation Cray OpenSHMEMX. We provide brief design overview, along with a review of functional and performance differences in Cray OpenSHMEMX comparing against the existing Cray SHMEM implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. 1.

    Though SMAT layer supports different communication layers, we use DMAPP and XPMEM in this work as others are experimental and not available in the current released Cray OpenSHMEMX version 8.0.1.

  2. 2.

    Number of Processing Elements (PEs) Per Node.


  1. GASNet: a portable high-performance communication layer for global address-space languages.

  2. OpenSHMEM standard version-1.3., a

  3. OpenSHMEM standard version-1.4., b

  4. OSU Micro-benchmarks.

  5. Cavium Thunder X2 ARM Processors.

  6. Global arrays: a portable shared memory model for distributed memory computers (1994)

    Google Scholar 

  7. Open MPI: a flexible high performance MPI (2005)

    Google Scholar 

  8. Almasi, G.: In: Padua, D.A. (ed.) Encyclopedia of Parallel Computing (2011)

    Google Scholar 

  9. Balaji, P., et al.: PMI: a scalable parallel process-management interface for extreme-scale systems. In: Keller, R., Gabriel, E., Resch, M., Dongarra, J. (eds.) EuroMPI 2010. LNCS, vol. 6305, pp. 31–41. Springer, Heidelberg (2010).

    Chapter  Google Scholar 

  10. Chapman, B., et al.: Introducing OpenSHMEM: SHMEM for the PGAS Community. In: Proceedings of the Fourth Conference on Partitioned Global Address Space Programming Model, PGAS 2010 (2010)

    Google Scholar 

  11. Doweck, J., et al.: Inside 6th-generation Intel Core: new microarchitecture code-named Skylake. IEEE Micro (2017)

    Google Scholar 

  12. Message Passing Forum. MPI: a message-passing interface standard. Technical report (1994)

    Google Scholar 

  13. Gropp, W.: MPICH2: a new start for MPI implementations. In: Kranzlmüller, D., Volkert, J., Kacsuk, P., Dongarra, J. (eds.) EuroPVM/MPI 2002. LNCS, vol. 2474, pp. 7–7. Springer, Heidelberg (2002).

    Chapter  Google Scholar 

  14. Hammarlund, P., et al.: Haswell: the fourth-generation intel core processor. IEEE Micro 34, 6–20 (2014)

    Article  Google Scholar 

  15. Kandalla, K., et al.: Current state of the cray MPT software stacks on the cray XC series supercomputers. In: Cray User Group (CUG) meeting 2017 (2017)

    Google Scholar 

  16. Knaak, D., Namashivayam, N.: Proposing OpenSHMEM extensions towards a future for hybrid programming and heterogeneous computing. In: Gorentla Venkata, M., Shamis, P., Imam, N., Lopez, M.G. (eds.) OpenSHMEM 2014. LNCS, vol. 9397, pp. 53–68. Springer, Cham (2015).

    Chapter  Google Scholar 

  17. Nalamalpu, A., et al.: Broadwell: a family of IA 14nm processors. In: Symposium on VLSI Circuits (VLSI Circuits) (2015)

    Google Scholar 

  18. Namashivayam, N., Knaak, D., Cernohous, B., Radcliffe, N., Pagel, M.: An evaluation of thread-safe and contexts-domains features in cray SHMEM. In: Gorentla Venkata, M., Imam, N., Pophale, S., Mintz, T.M. (eds.) OpenSHMEM 2016. LNCS, vol. 10007, pp. 163–180. Springer, Cham (2016).

    Chapter  Google Scholar 

  19. OpenMP Architecture Review Board. OpenMP application program interface version 4.5, November 2015.

  20. Sodani, S.A., et al.: Knights landing: second-generation Intel Xeon Phi product (2016)

    Google Scholar 

  21. ten Bruggencate, M., Roweth, D.: DMAPP: an API for one-sided programming model on baker systems. Technical report, Cray Users Group (CUG), August 2010

    Google Scholar 

  22. ten Bruggencate, M., Roweth, D., Oyanagi, S.: Thread-safe SHMEM extensions. In: Poole, S., Hernandez, O., Shamis, P. (eds.) OpenSHMEM 2014. LNCS, vol. 8356, pp. 178–185. Springer, Cham (2014).

    Chapter  Google Scholar 

Download references

Acknowledgment and Disclaimer

The authors would like to thank all the members of Cray Message Passing Toolkit for their involvement in the design and implementation of the Cray OpenSHMEMX software stack. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of associated organizations.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Naveen Namashivayam .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Namashivayam, N., Cernohous, B., Pou, D., Pagel, M. (2019). Introducing Cray OpenSHMEMX - A Modular Multi-communication Layer OpenSHMEM Implementation. In: Pophale, S., Imam, N., Aderholdt, F., Gorentla Venkata, M. (eds) OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Extreme Heterogeneity. OpenSHMEM 2018. Lecture Notes in Computer Science(), vol 11283. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-04917-1

  • Online ISBN: 978-3-030-04918-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics