Abstract
With rapid advancements in Ethernet technology, Ethernet speeds have increased by 10 fold, from 1 to 10Gbps, in a period of 2-3 years. This sudden increase in speeds has outpaced the rate at which processor and memory speeds have been increasing, raising concerns that TCP/IP processing will not scale to these levels. As a result, applications running on commercial servers will not be able to take advantage of the increased Ethernet bandwidth. This has led to a flurry of activity in the industry and academia focused on finding ways to scale up TCP/IP processing to 10Gbps and beyond. In this paper, we propose a novel technique called "Receive Side Coalescing" (RSC) that increases TCP/IP processing efficiencies significantly. RSC allows NICs to identify packets that belong to same TCP/IP flow and coalesce them into a single large packet. As a result, TCP/IP stack has to process fewer packets reducing per packet processing costs. NIC can do this coalescing of packets during interrupt moderation time hence packet latency is not affected. We have collected packet traces and analyzed those to find out how much coalescing is possible in different scenarios. Our analysis shows that about 50% reduction in number of packets is possible. We have prototyped RSC on Windows and Linux to understand the benefits, and the results show that 2-7% of savings in CPU utilization is possible at 1Gbps speeds. Projection models developed to estimate processing costs at 10Gbps show that RSC can save up to 20% of the CPU.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Alacritech SLIC: A Data Path TCP Offload methodology, http://www.alacritech.com/html/techreview.html
Blanton, E., Allman, M.: On the Impact of Bursting on TCP Performance. In: Proceedings of the Workshop for Passive and Active Measurement (March 2005)
Chase, J., et al.: End System Optimizations for High-Speed TCP. IEEE Communications, Special Issue on High-Speed TCP (2000)
Clark, D.D., Romkey, J., Salwen, H.: An analysis of TCP processing overhead. IEEE Communications 27(6), 23–29 (1989)
Ethereal Network Protocol Analyzer, http://www.ethereal.com/
Foong, A.P., Huff, T.R., Hum, H.H., Patwardhan, J.P., Regnier, G.J.: TCP Performance re-visited. In: Proc. IEEE Int. Symp. on Performance of Systems and Software, Austin, pp. 70–79 (March 2003)
Huggahalli, R., Iyer, R., Tetrick, S.: Direct Cache Access for High Bandwidth Network I/O. In: 32nd Annual International Symposium on Computer Architecture (ISCA 2005) (June 2005)
iSCSI, IP Storage Working Group, Internet Draft, available at: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-20.txt
Jain, R., Routhier, S.A.: Packet Trains - Measurements and a NewModel for Computer Network Traffic. IEEE Journal on Selected Areas in Communications 4, 986–994 (1986)
Jiang, H., Dovrolis, C.: Source-Level Packet Bursts: Causes and Effects. In: Proceedings Internet Measurement Conference (IMC) (October 2003)
Kay, J., Pasquale, J.: The importance of non-data touching processing overheads in TCP/IP. In: Proc. ACM SIGCOMM, San Francisco, pp. 259–268 (October 1993)
RDMA Consortium [Online], Available at: http://www.rdmaconsortium.org/
Kurmann, C., Müller, M., Rauch, F., Stricker, T.M.: Speculative defragmentation— A technique to improve the communication software efficiency for gigabit Ethernet. In: Proc. 9th IEEE Symp. High Performance Distr. Comp., Pittsburgh (August 2000)
Makineni, S., Iyer, R.: Architectural Characterization of TCP/IP Packet Processing on the Pentium M microprocessor. In: Int’l Conf. on High Performance Computer Architecture (HPCA-10) (Fecruary 2004)
McConnell, D.: IP Storage: A Technology Overview, http://www.dell.com/us/en/biz/top-ics/vectors_2001-ipstorage.htm
Mogul, J.C.: Observing TCP Dynamics in Real Networks. In: ACM SIGCOMM, pp. 305–317 (1992)
Mogul, J.: TCP offload is a dumb idea whose time has come. In: A Symposium on Hot Operating Systems (HOT OS) (2003)
Postel, J. (ed.): Internet Protocol - DARPA Internet program protocol specification, RFC 791 (September 1981)
Postel, J.B.: Transmission Control Protocol, RFC 793, Information Sciences Institute (September 1981)
Rangarajan, M., et al.: TCP Servers: Offloading TCP/IP Processing in Internet Servers. Design, Implementation, and Performance, Rutgers University, Technical Report, DCS-TR-481 (March 2002)
Regnier, G., Makineni, S., Illikkal, R., Iyer, R., et al.: TCP onloading for data center servers. IEEE Computer 37(11), 48–58 (2004)
Sinha, S., Katula, S., Katabi, D.: Harnessing TCP’s Burstiness with Flowlet Switching. In: 3rd ACM SIGCOMM Workshop on Hot Topics in Networks (HotNets III), San Diego, CA (November 2004)
The TTTCP Benchmark, http://ftp.arl.mil/~mike/ttcp.html
Yates, D.: Connection-Level Parallelism for Network Protocols on Shared-Memory Multi-processor Servers, Ph.D. Dissertation, University of Massachusetts, Amherst (1997)
Zhao, L., Makineni, S., Illikkal, R., Newell, D., Bhuyan, L.: TCP/IP Cache Characterization in Commercial Server Workloads. In: Seventh Workshop on Computer Architecture Evaluation using Commercial Workloads (CAECW-7), along with HPCA-10 (February 2004)
Chariot – More information, available at: http://www.ixiacom.com/solutions/display?skey=ixchariot
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Makineni, S. et al. (2006). Receive Side Coalescing for Accelerating TCP/IP Processing. In: Robert, Y., Parashar, M., Badrinath, R., Prasanna, V.K. (eds) High Performance Computing - HiPC 2006. HiPC 2006. Lecture Notes in Computer Science, vol 4297. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11945918_31
Download citation
DOI: https://doi.org/10.1007/11945918_31
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-68039-0
Online ISBN: 978-3-540-68040-6
eBook Packages: Computer ScienceComputer Science (R0)