Abstract
Although extremely high-speed interconnects are available today, the traditional protocol stacks such as TCP/IP and UDP/IP are not able to utilize the maximum network bandwidth due to inherent overheads in the protocol stacks. Such overheads are a big obstacle for high-performance computing applications to exploit high-speed interconnects in cluster environments. To address this issue, many researchers have been presenting analyses of protocol overheads and suggesting a number of optimization approaches to harness the TCP/IP suite over high-speed interconnects. However, to the best of our knowledge, there is no study that analyzes and optimizes the protocol overheads thoroughly in an integrated manner. In this paper, we exploit a set of protocol optimization mechanisms in an integrated manner while dealing with the full spectrum of the protocol layers from the transport layer to the physical layer. To evaluate the impact of each protocol overhead, we apply the optimization mechanisms one by one and perform detailed analyses at each step. The thorough overhead measurements and analyses reveal the dependencies between protocol overheads. With our comprehensive optimizations, we show that UDP/IP can utilize more than 95% of the maximum network throughput a Myrinet-based experimental system can provide.
Similar content being viewed by others
References
Anderson D, Chase J, Gadde S, Gallatin A, Yocum K, Feeley M (1998) Cheating the I/O bottleneck: network storage with trapeze/myrinet. In: Proceedings of the 1998 USENIX technical conference, June 1998
Anderson TE, Culler DE, Patterson DA, NOW Team (1995) A case for networks of workstations: NOW. IEEE Micro 15(1):54–64
Arnould E, Bitz F, Cooper E, Kung HT, Sansom R, Steenkiste P (1989) The design of nectar: a network backplane for heterogeneous multicomputers. In: Third international conference on architectural support for programming languages and operating systems, April 1989, pp 205–216
Balaji P, Narravula S, Vaidyanathan K, Krishnamoorthy S, Wu J, Panda DK (2004) Sockets direct protocol over InfiniBand in clusters: is it beneficial? IEEE international symposium on performance analysis of systems and software (ISPASS 04), March 2004
Balaji P, Shah HV, Panda DK (2004) Sockets vs RDMA interface over 10-Gigabit networks: an in-depth analysis of the memory traffic bottleneck. In: Workshop on remote direct memory access (rdma): applications, implementations, and technologies (RAIT 2004), September 2004
Banks D, Prudence M (1993) A high-performance network architecture for a PA-RISC workstation. IEEE J Sel Areas Commun 11(2):191–202
Barak A, Gilderman I, Metrik I (1999) Performance of the communication layers of TCP/IP with the myrinet gigabit LAN. Comput Commun 22:989–997
Boden NJ, Cohen D, Felderman RE, Kulawik AE, Seitz CL, Seizovic JN, Su W-K (1995) Myrinet—a gigabit-per-second local-area network. IEEE Micro 15(1):29–36
Braden R, Borman D, Partridge C (1988) Computing the Internet checksum. In: IETF networking working group request for comments: 1071, September 1988
Brustoloni JC, Steenkiste P (1997) Copy emulation in checksummed, multiple-packet communication. In: IEEE Infocom ’97, April 1997
Chase J, Gallatin A, Yocum K (2001) End-system optimizations for high-speed TCP. IEEE Commun Mag 39(4):68–75
Chu HJ (1996) Zero-copy TCP in solaris. In: 1996 Winter USENIX, 1996
Clark D, Jacobson V, Romkey J, Salwen H (1989) An analysis of TCP processing overhead. IEEE Commun Mag 27(6):23–29
Dalton C, Watson G, Banks D, Calamvokis C, Edwards A, Lumley J (1993) Afterburner. IEEE Netw 7(4):36–43
Druschel P, Peterson LL (1993) Fbufs: a high-bandwidth cross-domain transfer facility. In: Proceedings of 14th ACM SOSP, 1993, pp 189–202
Dunning D, Regnier G, McAlpine G, Cameron D, Shubert B, Berry AM, Gronke E, Dodd C (1998) The virtual interface architecture. IEEE Micro 8:66–76
von Eicken T, Basu A, Buch V, Vogels W (1995) U-Net: a user-level network interface for parallel and distributed computing. In: Proceedings of 15th ACM SOSP, December 1995, pp 40–53
Feng W-C, Hurwitz J, Newman H, Ravot S, Cottrell RL, Martin O, Coccetti F, Jin C, Wei X, Low S (2003) Optimizing 10-gigabit Ethernet for networks of workstations, clusters, and grids: a case study. In: Proceedings of SC2003: high-performance networking and computing conference, November 2003
Foster I, Kesselman C (1999) The grid: blueprint for a new computing infrastructure. Morgan Kaufmann, San Francisco
Gallatin A, Chase J, Yocum K (1999) Trapeze/IP: TCP/IP at near-gigabit speeds. In: Proceedings of 1999 USENIX technical conference (freenix track), June 1999
Gigabit Ethernet Alliance (1997) IEEE 802.3z. The Emerging Gigabit Ethernet Standard
10 Gigabit Ethernet Alliance (2002) Introduction to TCP/IP offload engine (TOE), Version 1.0, April 2002
Halstead DM, Bode B, Turner D, Lewis V (1999) Giga-plant scalable cluster. In: Proceedings of the USENIX extreme Linux technical conference, June 1999, pp 10–15
IEEE (2002) IEEE Std 802.3ae-2002, Media access control (MAC) Parameters, physical layers, and management parameters for 10 Gbps operation, August 2002
InfiniBand Trade Association. http://www.infinibandta.org
Jin H-W, Balaji P, Yoo C, Choi J-Y, Panda DK (2005) Exploiting NIC architectural support for enhancing IP based protocols on high performance networks. J Parallel Distrib Comput 65(11):1348–1365
Jin H-W, Yoo C, Choi J-Y (2003) Firmware-level latency analysis on a gigabit network. J Supercomput 26(1):59–75
Kay J, Pasquale J (1996) Profiling and reducing processing overheads in TCP/IP. IEEE/ACM Trans Netw 4(6):817–828
Kurmann C, Muller M, Rauch F, Stricker T (2000) Speculative defragmentation a technique to improve the communication software efficiency for gigabit ethernet. In: Proceedings of 9th IEEE symposium on high performance distributed computing (HPDC), August 2000
Makineni S, Iyer R (2004) Architectural characterization of TCP/IP packet processing on the Pentium M microprocessor. In: Proceedings of 10th international symposium on high performance computer architecture (HPCA-10), February 2004, pp 152–161
Massalin H, Pu C (1989) Threads and input/output in the synthesis kernel. In: Proceedings of ACM symposium on operating systems principles (SOSP), 1989, pp 191–201
McKenney P, Dove K (1992) Efficient demultiplexing of incoming TCP packets. In: Proceedings of ACM SIGCOMM ’92, August 1992
Mogul J (1991) Network locality at the scale of pro cessors. In: Proceedings of ACM SIGCOMM ’91, September 1991
Myricom Inc (1996) Myrinet user’s guide. http://www.myri.com
Myricom Inc (1999) LANai 4. http://www.myri.com, February 1999
Myricom Inc (2000) The GM message passing system. http://www.myri.com, January 2000
Myricom Inc (2001) PCI64 programmer’s documentation. http://www.myri.com, May 2001
Pakin S, Lauria M, Chien A (1995) High performance messaging on workstations: Illinois fast messages (FM) for Myrinet. In: Proceedings of SC95, 1995
Partridge C, Pink S (1993) A faster UDP. IEEE/ACM Trans Netw 1(4):429–440
Prylli L, Tourancheau B (1998) BIP: a new protocol designed for high performance networking on myrinet. In: Proceedings of the international parallel processing symposium workshop on personal computer based networks of workstations, 1998
Romanow A, Bailey S (2003) An overview of RDMA over IP. In: Proceedings on first international workshop on protocols for fast long-distance networks (PFLDnet2003), February 2003
Shah H, Minturn D, Foong A, McAlpine G, Madukkarumukumana R, Regnier G (2001) CSP: a novel system architecture for scalable Internet and communication services. In: Proceedings of USENIX symposium on internet technologies and systems (USITS01), 2001
Shivam P, Wyckoff P, Panda DK (2002) Can user level protocols take advantage of Multi-CPU NICs? In: Proceedings of international parallel and distributed processing symposium (IPDPS ’02), April 2002
Smith J, Traw C (1993)Giving applications access to Gb/s networking. IEEE Netw 7(4):44–52
USNA (1984) TTCP: a test of TCP and UDP performance, December 1984
Willmann P, Kim H-Y, Rixner S, Pai V (2005) An efficient programmable 10 gigabit ethernet network interface card. In: Proceedings of HPCA-11, February 2005
Wolman A, Voelker G, Thekkath C (1994) Latency analysis of TCP on an ATM network. In: 1994 Winter USENIX, January 1994
Woodside CM, Montealegre JR (1989) The effect of buffering strategies on protocol execution performance. IEEE Trans Commun 37(6): 545–554
Yoo C, Jin H-W, Kwon S-C (2001) Asynchronous UDP. IEICE Trans Commun E84-B(12):3243–3251
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by the Faculty Research Fund of Konkuk University in 2006.
Rights and permissions
About this article
Cite this article
Jin, HW., Yoo, C. Impact of protocol overheads on network throughput over high-speed interconnects: measurement, analysis, and improvement. J Supercomput 41, 17–40 (2007). https://doi.org/10.1007/s11227-007-0101-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-007-0101-0