Abstract
In this paper we describe the architecture, design, and performance of the new cluster switch fabric and adapter called HPS (High Performance Switch). HPS delivers very low latency and very high bandwidth. We demonstrate latency of less than 4.3us MPI library; 1.8GB/s of delivered unidirectional bandwidth and 2.9GB/s of bidirectional bandwidth between 2 MPI tasks running on 1.9GHz Power 4+ IH based nodes. HPS also supports RDMA (remote direct memory access capability). A unique capability of RDMA over HPS is that reliable RDMA is supported over an underlying unreliable transport (unlike Infiniband and other RDMA transport protocols which depend on the underlying transport being reliable). We profile the performance of RDMA and its impact on striping for systems in which multiple network adapters are available to tasks of parallel jobs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
[DGSMP] Treumann, R.: DGSM: Data Gather Scatter Machine. IBM Internal Report
[FPGS] Frye, D., Gildea, K., Hochschild, P., Snir, M.: The communication software and Parallel Environment for the IBM SP2. IBM Systems Journal 34(2), 205–221 (1995)
[GA] Nieplocha, J., Ju, J., Krishnan, M.K., Palmer, B., Tipparaju, V.: The Global Arrays User Manual, http://www.emsl.pnl.gov/docs/global/user.html
[GPFSP] GPFS White Paper: http://www-1.ibm.com/servers/eserver/pseries/software/sp/gpfs.html
[ITAPI] IT-API: Open group consortium on API definition for RDMA capable networks, http://www.opengroup.org
[IBTA] Infiniband Architecture, http://www.infinibandta.org/ibta/
[LAPIP] IBM’s LAPI Documentation, http://rs6ktech.dfw.ibm.com/sp/docs/pssp3.4/pssphtml/cmdsv2/am0trmst02.html
[LAPIP2] Shah, G., Nieplocha, J., Mirza, J., Kim, C., Harrison, R.J., Govindaraju, R.K., Gildea, K., DiNicola, P., Bender, C.A.: Performance and Experience with LAPI – A New High Performance Communication Library for the IBM RS/6000 SP. In: Proceedings of IPPS (International Parallel Processing Symposium) (1998)
[MPILAPIP] Banikazemi, M., Govindaraju, R.K., Blackmore, R., Panda, D.B.: MPI-LAPI: An Efficient implementation of MPI for RS/6000 SP Systems. IEEE Transactions for Parallel and Distributed Computing 12(10), 1081–1093 (2001)
[RDMAP] An Efficient reliable RDMA mechanism over an unreliable network transport protocol. IBM Patent (submitted) (April 2004)
[RDMAX] Bode, B.M., Hill, J.J., Benjegerdes, T.R.: Cluster Interconnect Overview. In: USENIX 2004 (2004)
[REVIB] Benjegerdes, T.R., Bode, B.M.: Infiniband Performance Review. In: USENIX 2004 (2004)
[VIBPCIEX] Liu, J., Mamidala, A., Vishnu, A., Panda, D.K.: Performance Evaluation of Infiniband with PCI Express. Hot Interconnect 12 (August 2004)
[VMPI] Liu, J., Wu, J., Kini, S.P., Wyckoff, P., Panda, D.K.: High Performance RDMA-Based MPI implementation over Infiniband. In: 17th International Conference on Supercomputing (June 2003)
[YNET] Bell, C., Bonachea, D., Cote, Y., Duell, J., Hargrove, P., Husbands, P., Iancu, C., Welcome, M., Yelick, K.: An evaluation of current high-performance networks. In: International Parallel and Distributed Processing Symposium (April 2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Govindaraju, R.K. et al. (2004). Architecture and Early Performance of the New IBM HPS Fabric and Adapter. In: Bougé, L., Prasanna, V.K. (eds) High Performance Computing - HiPC 2004. HiPC 2004. Lecture Notes in Computer Science, vol 3296. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30474-6_21
Download citation
DOI: https://doi.org/10.1007/978-3-540-30474-6_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-24129-4
Online ISBN: 978-3-540-30474-6
eBook Packages: Computer ScienceComputer Science (R0)