Architecture and Early Performance of the New IBM HPS Fabric and Adapter

  • Rama K Govindaraju
  • Peter Hochschild
  • Don Grice
  • Kevin Gildea
  • Robert Blackmore
  • Carl A Bender
  • Chulho Kim
  • Piyush Chaudhary
  • Jason Goscinski
  • Jay Herring
  • Steven Martin
  • John Houston
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3296)

Abstract

In this paper we describe the architecture, design, and performance of the new cluster switch fabric and adapter called HPS (High Performance Switch). HPS delivers very low latency and very high bandwidth. We demonstrate latency of less than 4.3us MPI library; 1.8GB/s of delivered unidirectional bandwidth and 2.9GB/s of bidirectional bandwidth between 2 MPI tasks running on 1.9GHz Power 4+ IH based nodes. HPS also supports RDMA (remote direct memory access capability). A unique capability of RDMA over HPS is that reliable RDMA is supported over an underlying unreliable transport (unlike Infiniband and other RDMA transport protocols which depend on the underlying transport being reliable). We profile the performance of RDMA and its impact on striping for systems in which multiple network adapters are available to tasks of parallel jobs.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    [DGSMP] Treumann, R.: DGSM: Data Gather Scatter Machine. IBM Internal ReportGoogle Scholar
  2. 2.
    [FPGS] Frye, D., Gildea, K., Hochschild, P., Snir, M.: The communication software and Parallel Environment for the IBM SP2. IBM Systems Journal 34(2), 205–221 (1995)CrossRefGoogle Scholar
  3. 3.
    [GA] Nieplocha, J., Ju, J., Krishnan, M.K., Palmer, B., Tipparaju, V.: The Global Arrays User Manual, http://www.emsl.pnl.gov/docs/global/user.html
  4. 4.
  5. 5.
    [ITAPI] IT-API: Open group consortium on API definition for RDMA capable networks, http://www.opengroup.org
  6. 6.
    [IBTA] Infiniband Architecture, http://www.infinibandta.org/ibta/
  7. 7.
  8. 8.
    [LAPIP2] Shah, G., Nieplocha, J., Mirza, J., Kim, C., Harrison, R.J., Govindaraju, R.K., Gildea, K., DiNicola, P., Bender, C.A.: Performance and Experience with LAPI – A New High Performance Communication Library for the IBM RS/6000 SP. In: Proceedings of IPPS (International Parallel Processing Symposium) (1998)Google Scholar
  9. 9.
    [MPILAPIP] Banikazemi, M., Govindaraju, R.K., Blackmore, R., Panda, D.B.: MPI-LAPI: An Efficient implementation of MPI for RS/6000 SP Systems. IEEE Transactions for Parallel and Distributed Computing 12(10), 1081–1093 (2001)CrossRefGoogle Scholar
  10. 10.
    [RDMAP] An Efficient reliable RDMA mechanism over an unreliable network transport protocol. IBM Patent (submitted) (April 2004)Google Scholar
  11. 11.
    [RDMAX] Bode, B.M., Hill, J.J., Benjegerdes, T.R.: Cluster Interconnect Overview. In: USENIX 2004 (2004)Google Scholar
  12. 12.
    [REVIB] Benjegerdes, T.R., Bode, B.M.: Infiniband Performance Review. In: USENIX 2004 (2004)Google Scholar
  13. 13.
    [VIBPCIEX] Liu, J., Mamidala, A., Vishnu, A., Panda, D.K.: Performance Evaluation of Infiniband with PCI Express. Hot Interconnect 12 (August 2004)Google Scholar
  14. 14.
    [VMPI] Liu, J., Wu, J., Kini, S.P., Wyckoff, P., Panda, D.K.: High Performance RDMA-Based MPI implementation over Infiniband. In: 17th International Conference on Supercomputing (June 2003)Google Scholar
  15. 15.
    [YNET] Bell, C., Bonachea, D., Cote, Y., Duell, J., Hargrove, P., Husbands, P., Iancu, C., Welcome, M., Yelick, K.: An evaluation of current high-performance networks. In: International Parallel and Distributed Processing Symposium (April 2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Rama K Govindaraju
    • 1
  • Peter Hochschild
    • 2
  • Don Grice
    • 1
  • Kevin Gildea
    • 1
  • Robert Blackmore
    • 1
  • Carl A Bender
    • 1
  • Chulho Kim
    • 1
  • Piyush Chaudhary
    • 1
  • Jason Goscinski
    • 1
  • Jay Herring
    • 1
  • Steven Martin
    • 1
  • John Houston
    • 1
  1. 1.Server Development LabIBMPoughkeepsieUSA
  2. 2.T. J. Watson Research CenterIBM ResearchHawthorneUSA

Personalised recommendations