A High Performance Superpipeline Protocol for InfiniBand

  • Alexandre Denis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6853)


InfiniBand high performance networks require that the buffers used for sending or receiving data are registered. Since memory registration is an expensive operation, some communication libraries use caching (rcache) to amortize its cost, and copy data into pre-registered buffers for small messages. In this paper, we present a software protocol for InfiniBand that always uses a memory copy, and amortizes the cost of this copy with a superpipeline to overlap the memory copy and the RDMA.We propose a performance model of our protocol to study its behavior and optimize its parameters. We have implemented our protocol in the NewMadeleine communication library. The results of MPI benchmarks show a significant improvement in cache-unfriendly applications that do not reuse the same memory blocks all over the time, without degradation for cache-friendly applications.


Receiver Side Cluster Graphene Message Length Chunk Size Communication Library 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mietke, F., Rex, R., Baumgartl, R., Mehlan, T., Hoefler, T., Rehm, W.: Analysis of the memory registration process in the mellanox infiniband software stack. In: Nagel, W.E., Walter, W.V., Lehner, W. (eds.) Euro-Par 2006. LNCS, vol. 4128, pp. 124–133. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  2. 2.
    Liu, J., Jiang, W., Wyckoff, P., Panda, D.K., Ashton, D., Gropp, W., Buntinas, D., Toonen, B.: Design and implementation of MPICH2 over infiniband with RDMA support. In: Proceedings of the International Parallel and Distributed Processing Symposium, IPDPS 2004 (2004)Google Scholar
  3. 3.
    Tezuka, H., O’Carroll, F., Hori, A., Ishikawa, Y.: Pin-down cache: a virtual memory management technique for zero-copy communication. In: Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing (IPPS/SPDP), pp. 308–314 (April 1998)Google Scholar
  4. 4.
    Ohio, P.W., Wyckoff, P., Wu, J.: Memory registration caching correctness. In: IEEE International Symposium on Cluster Computing and the Grid(CCGrid). IEEE Computer Society, Los Alamitos (2005)Google Scholar
  5. 5.
    Culler, D., Karp, R., Patterson, D., Sahay, A., Schauser, K.E., Santos, E., Subramonian, R., von Eicken, T.: Logp: towards a realistic model of parallel computation. In: ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. PPOPP 1993, pp. 1–12. ACM, New York (1993)Google Scholar
  6. 6.
    Gupta, R., Tipparaju, V., Nieplocha, J., Panda, D.K.: Efficient barrier using remote memory operations on via-based clusters. In: Cluster 2002 (September 2002)Google Scholar
  7. 7.
    Aumage, O., Brunet, E., Furmento, N., Namyst, R.: Newmadeleine: a fast communication scheduling engine for high performance networks. In: CAC 2007: Workshop on Communication Architecture for Clusters, Held in Conjunction with IPDPS 2007 (March 2007)Google Scholar
  8. 8.
    Trahay, F., Denis, A., Aumage, O., Namyst, R.: Improving reactivity and communication overlap in MPI using a generic I/O manager. In: EuroPVM/MPI. LNCS, Springer, Heidelberg (2007)Google Scholar
  9. 9.
    Hoefler, T., Mehlan, T., Mietke, F., Rehm, W.: Logfp – a model for small messages in infiniband. In: IPDPS (2006)Google Scholar
  10. 10.
    Bell, C., Bonachea, D.: A new dma registration strategy for pinning-based high performance networks. In: Proceedings of the International Symposium on Parallel and Distributed Processing, p. 10 (2003)Google Scholar
  11. 11.
    Ou, L., He, X., Han, J.: An efficient design for fast memory registration in RDMA. Journal of Network and Computer Applications 32, 641–642 (2009)CrossRefGoogle Scholar
  12. 12.
    Woodall, T.S., Shipman, G.M., Bosilca, G., Graham, R.L., Maccabe, A.B.: High performance RDMA protocols in HPC. In: Mohr, B., Träff, J.L., Worringen, J., Dongarra, J. (eds.) PVM/MPI 2006. LNCS, vol. 4192, pp. 76–85. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  13. 13.
    Goldenberg, D., Kagan, M., David, R., Tsirkin, M.S.: Zero copy sockets direct protocol over infiniband – preliminary implementation and performance analysis. In: Symposium on High-Performance Interconnects, HOTI 2005 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Alexandre Denis
    • 1
  1. 1.INRIA Bordeaux Sud-Ouest / LaBRIFrance

Personalised recommendations