NIC-Assisted Cache-Efficient Receive Stack for Message Passing over Ethernet

  • Brice Goglin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5704)


High-speed networking in clusters usually relies on advanced hardware features in the NICs, such as zero-copy. Open-MX is a high-performance message passing stack designed for regular Ethernet hardware without such capabilities.

We present the addition of multiqueue support in the Open-MX receive stack so that all incoming packets for the same process are treated on the same core. We then introduce the idea of binding the target end process near its dedicated receive queue. This model leads to a more cache-efficient receive stack for Open-MX. It also proves that very simple and stateless hardware features may have a significant impact on message passing performance over Ethernet.

The implementation of this model in a firmware reveals that it may not be as efficient as some manually tuned micro-benchmarks. But our multiqueue receive stack generally performs better than the original single queue stack, especially on large communication patterns where multiple processes are involved and manual binding is difficult.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bailey, D.H., Barszcz, E., Barton, J.T., Browning, D.S., Carter, R.L., Dagum, D., Fatoohi, R.A., Frederickson, P.O., Lasinski, T.A., Schreiber, R.S., Simon, H.D., Venkatakrishnan, V., Weeratunga, S.K.: The NAS Parallel Benchmarks. The International Journal of Supercomputer Applications 5(3), 63–73 (Fall 1991)CrossRefGoogle Scholar
  2. 2.
    Browne, S., Dongarra, J., Garner, N., Ho, G., Mucci, P.: A Portable Programming Interface for Performance Evaluation on Modern Processors. The International Journal of High Performance Computing Applications 14(3), 189–204 (2000)CrossRefGoogle Scholar
  3. 3.
    Mellanox ConnectX - 4th Generation Server & Storage Adapter Architecture,
  4. 4.
    FCoE (Fibre Channel over Ethernet),
  5. 5.
    Gabriel, E., Fagg, G.E., Bosilca, G., Angskun, T., Dongarra, J.J., Squyres, J.M., Sahay, V., Kambadur, P., Barrett, B., Lumsdaine, A., Castain, R.H., Daniel, D.J., Graham, R.L., Woodall, T.S.: Open MPI: Goals, concept, and design of a next generation MPI implementation. In: Proceedings, 11th European PVM/MPI Users’ Group Meeting, Budapest, Hungary, September 2004, pp. 97–104 (2004)Google Scholar
  6. 6.
    Goglin, B.: Design and Implementation of Open-MX: High-Performance Message Passing over generic Ethernet hardware. In: CAC 2008: Workshop on Communication Architecture for Clusters, held in conjunction with IPDPS 2008, Miami, FL, April 2008. IEEE Computer Society Press, Los Alamitos (2008)Google Scholar
  7. 7.
    Goglin, B.: Improving Message Passing over Ethernet with I/OAT Copy Offload in Open-MX. In: Proceedings of the IEEE International Conference on Cluster Computing, Tsukuba, Japan, September 2008, pp. 223–231. IEEE Computer Society Press, Los Alamitos (2008)Google Scholar
  8. 8.
    Grossman, L.: Large Receive Offload Implementation in Neterion 10GbE Ethernet Driver. In: Proceedings of the Linux Symposium (OLS 2005), Ottawa, Canada, July 2005, pp. 195–200 (2005)Google Scholar
  9. 9.
    Huggahalli, R., Iyer, R., Tetrick, S.: Direct Cache Access for High Bandwidth Network I/O. SIGARCH Computer Architecture News 33(2), 50–59 (2005)CrossRefGoogle Scholar
  10. 10.
  11. 11.
    Myricom Myri-10G,
  12. 12.
    Myricom, Inc. Myrinet Express (MX): A High Performance, Low-Level, Message-Passing Interface for Myrinet (2006),
  13. 13.
    The Parallel Virtual File System, version 2,
  14. 14.
    Mohammad, J.R., Afsahi, A.: 10-Gigabit iWARP Ethernet: Comparative Performance Analysis with Infiniband and Myrinet-10G. In: Proceedings of the International Workshop on Communication Architecture for Clusters (CAC), held in conjunction with IPDPS 2007, Long Beach, CA, March 2007, p. 234 (2007)Google Scholar
  15. 15.
    Santos, J.R., Turner, Y., Janakiraman, G(J.), Pratt, I.: Bridging the Gap between Software and Hardware Techniques for I/O Virtualization. In: Proceedings of USENIX 2008 Annual Technical Conference, Boston, MA, June 2008, pp. 29–42 (2008)Google Scholar
  16. 16.
    Willmann, P., Rixner, S., Cox, A.L.: An Evaluation of Network Stack Parallelization Strategies in Modern Operating Systems. In: Proceedings of the USENIX Technical Conference, Boston, MA, pp. 91–96 (2006)Google Scholar
  17. 17.
    Yi, Z., Waskiewicz, P.P.: Enabling Linux Network Support of Hardware Multiqueue Devices. In: Proceedings of the Linux Symposium (OLS 2007), Ottawa, Canada, June 2007, pp. 305–310 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Brice Goglin
    • 1
  1. 1.INRIA BordeauxFrance

Personalised recommendations