Skip to main content
SpringerLink
Log in

Communication optimization for RDMA-based science data transmission tools

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Big data has raised new challenges to data communication and transmission capacity. RDMA (Remote Direct Memory Access) Transport Protocol can reduce the communication delay of big data through kernel memory bypass technology and zero-copy technology. This paper introduced RDMA technology for the network performance optimization of BBCP (Babar Copy Program), a big data network file copy tool. Also, transmission queue-oriented Tetris Scheduler algorithm, I/O performance optimization and buffer memory optimization technique were introduced to realize RDMA-based file transfer tool, OBCP (Optimized Babar Copy Program). The experiment results verified that the optimization method could decrease the file transmission time to one-tenth of original BBCP and improve the performance by 50 % compared with related works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Martin H, López P (2011) The world’s technological capacity to store, communicate, and compute information. Science 332(6025):60–65

    Article  Google Scholar 

  2. Li T, Binghang Y, Luan Z (2010) Fast large file distribution in data grid. In: Intelligent Computing and Integrated Systems (ICISS), 2010 International Conference on. IEEE (2010)

  3. http://www.rdmaconsortium.org/

  4. Rashti Mohammad J, Afsahi A (2009) A speculative and adaptive MPI Rendezvous protocol over RDMA-enabled interconnects. Int. J. Parallel Progr. 37(2):223–246

    Article  MATH  Google Scholar 

  5. Pinkerton J (2002) The case for RDMA. RDMA Consortium, p 27 May 29

  6. Shivam P, Wyckoff P, Panda D (2001) EMP: zero-copy OS-bypass NIC-driven gigabit ethernet message passing. In: Supercomputing, ACM/IEEE 2001 Conference. IEEE

  7. Hanushevsky A, Artem T, Les C (2001) Peer-to-peer computing for secure high performance data copying. In: Proc. of the 2001 Int. Conf. on Computing in High Energy and Nuclear Physics (CHEP 2001), Beijing

  8. Adam Smyk, Tudruj Marek (2006) RDMA control support for fine-grain parallel computations. J Syst Archit 52(2):117–128

    Article  Google Scholar 

  9. Wang B, Chen L, Liu Y (2013) The research of performance optimization technology in the high-speed network. In: Software Engineering and Service Science (ICSESS), 2013 4th IEEE International Conference on. IEEE

  10. IBCP. https://github.com/UniqueStudio/ibcp

  11. Taub. https://campuscluster.illinois.edu/hardware/#taub

  12. Weizhe Z, Zhang Y, Kim T-H (2014) Detecting bad information in mobile wireless networks based on the wireless application protocol. Computing 96(9):855–874

    Article  Google Scholar 

  13. Zhang W, Wang X, Lu B, Kim TH (2013) Secure encapsulation and publication of biological services in the cloud computing environment. BioMed Res Int 2013

  14. Zhang W, He H, Zhang Q, Kim TH (2014) PhoneProtector: protecting user privacy on the android-based mobile platform. Int J Distrib Sens Netw 2014

Download references

Acknowledgments

This work is supported by the National Basic Research Program of China (973 Program) under grant No. 2011CB302605, the National Science Foundation of China (NSFC) under grant No. 61173145, and also the Doctoral Program of Higher Education of China under grant No. 20132302110037.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, HL, China

    Weizhe Zhang, Meng Hao & Zhiyong Xu

Authors
  1. Weizhe Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meng Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhiyong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weizhe Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, W., Hao, M. & Xu, Z. Communication optimization for RDMA-based science data transmission tools. J Supercomput 72, 3312–3327 (2016). https://doi.org/10.1007/s11227-015-1399-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-015-1399-7

Keywords

Search

Navigation