Performance of the 25 Gbps/100 Gbps Fullmesh RoCE Network Using Mellanox ConnetX-4 Lx Adapter and Ruijie S6500 Ethernet Switch

  • Hui LiEmail author
  • Xiaolong Chen
  • Tao Song
  • Haiyin Chen
  • Hao Chen
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 927)


Zero-copy RDMA is promising in the support of IPC operations for a wide range of applications. Modern high-performance network protocols such as Infiniband, RoCE, or iWARP offer RDMA, which provides high bandwidth and low latency communication for applications. In this paper, we assess 25 Gbps/100 Gpbs fullmesh RoCE network using the Mellanox ConnetX-4 Lx adapter and Ruijie s6500 switch, which are already largely deployed in Alibaba, Baidu, and Tencent data center network environments. We find that the capability of the switch can be a bottleneck in the fullmesh network, and we also find that our fullmesh deployment can achieve linear scalability of throughput as long as the total throughput of servers does not exceed the link rate of switches.


  1. 1.
    Beck, M., Kagan, M.: Performance evaluation of the RDMA over Ethernet (RoCE) standard in enterprise data centers infrastructure. In: Proceedings of DC-CaVES (2011)Google Scholar
  2. 2.
    Guo, C., Wu, H., Deng, Z., Soni, G., Ye, J., Padhye, J., Lipshteyn, M.: RDMA over commodity ethernet at scale. In: SIGCOMM 2016 (2016)Google Scholar
  3. 3.
    Shpiner, A., Zahavi, E., Dahley, O., Barnea, A., Damsker, R., Yekelis, G., Zus, M., Kuta, E., Baram, D.: RoCE Rocks without PFC: Detailed EvaluationGoogle Scholar
  4. 4.
    Zhu, Y., et al.: Congestion control for large-scale RDMA deployments. In: ACM SIGCOMM (2015)Google Scholar
  5. 5.
    Islam, N.S., Rahman, M.W., Jose, J., Rajachandrasekar, R., Wang, H., Subgramoni, H., Murthy, C., Panda, D.K.: High-performance RDMA-based design of Hadoop mapreduce over Infiniband. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC 2012 (2012)Google Scholar
  6. 6.
    Lu, X., Islam, N.S., Wasi-Ur-Rahman, M., Jose, J., Subramoni, H.: High-performance design of Hadoop RPC with RDMA over InfiniBand. In: International Conference on Parallel Processing (2013)Google Scholar
  7. 7.
    Accelerating Spark with RDMA for Big Data Processing: Early ExperiencesGoogle Scholar
  8. 8.
    Shankar, D., Lu, X., Jose, J., Wasi-ur-Rahman, M., Islam, N., Panda, D.K.: Can RDMA benefit online data processing workloads on memcached and MySQL. ISPASS, Philadelphia, PA, USA (2015)Google Scholar
  9. 9.
  10. 10.
    How to implement the VSU-lite in DCN network.
  11. 11.
  12. 12.
    Mellanox OFED for Linux User ManualGoogle Scholar
  13. 13.
  14. 14.
  15. 15.
  16. 16.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Hui Li
    • 1
    Email author
  • Xiaolong Chen
    • 1
  • Tao Song
    • 1
  • Haiyin Chen
    • 1
  • Hao Chen
    • 1
  1. 1.Testing Group of the Data Center NetworkRuijie NetworkFuzhouChina

Personalised recommendations