International Journal of Parallel Programming

, Volume 37, Issue 6, pp 556–571 | Cite as

Architectural Breakdown of End-to-End Latency in a TCP/IP Network

  • Steen LarsenEmail author
  • Parthasarathy Sarangam
  • Ram Huggahalli
  • Siddharth Kulkarni


Adoption of the 10GbE Ethernet standard as a high performance interconnect has been impeded by two important performance-oriented considerations: (1) processing requirements of common protocol stacks and (2) end-to-end latency. The overheads of typical software based protocol stacks on CPU utilization and throughput have been well evaluated in several recent studies. We focus on end-to-end latency and present a detailed characterization across typical server system hardware and software stack components. We demonstrate that application level end-to-end one-way latency with a 10GbE connection can be as low as 10 μs for a single isolated request in a standard Linux network stack. The paper analyzes the components of the latency and discusses possible significant variations to the components under realistic conditions. We found that methods that optimize for throughput can significantly compromise Ethernet based latencies. Methods to pursue reducing the minimum latency and controlling the variations are presented.


Latency Network Communication Ethernet 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Foong, A., Huff, T., Hum, H., Patwardhan, J., Regnier, G.: TCP performance re-visited. In: Proceedings of the IEEE International Symposium on Performance of Systems & Software, Austin, Mar. 2003Google Scholar
  2. 2.
    Mogul, J.: TCP offload is a dumb idea whose time has come. In: Proceedings of the 9th Workshop on Hot Topics in Operating Systems (HotOS IX). Usenix Assoc. (2003)
  3. 3.
    Huggahalli, R., et al.: Direct cache access for high bandwidth network I/O. In: International Symposium on Computer Architecture (ISCA) (2005)
  4. 4.
    Regnier, G., et. al.: TCP onloading for data center servers. In: IEEE Computer, Nov. 2004Google Scholar
  5. 5.
  6. 6.
  7. 7.
  8. 8.
    Feng, W., Balaji, P., Baron, C., Bhuyan, L.N., Panda, D.K.: Performance characterization of a 10-Gigabit Ethernet TOE; high performance interconnects. In: Proceedings of the 13th Symposium, Aug. 2005.
  9. 9.
    Express Base Specification Revision, P.C.I.: 2.0. (2006)
  10. 10.
  11. 11.
  12. 12.
    IA-32 Intel® architecture software developer’s manual volume 3: system programming guide appendix A, June 2005Google Scholar
  13. 13.
    Foong, A., Fung, J., Newell, D.: An in-depth analysis of the impact of processor affinity on network performance. In: Proceedings of the IEEE International Conference Networks, IEEE Press (2004)Google Scholar
  14. 14.
  15. 15.
  16. 16.
    Hansen, J., Jul, E.: Latency reduction using a polling scheduler. In: Proceedings of the Second Workshop on Cluster-Based Computing, ACM-SIGARCH (2000)Google Scholar
  17. 17.
  18. 18.

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Steen Larsen
    • 1
    Email author
  • Parthasarathy Sarangam
    • 1
  • Ram Huggahalli
    • 1
  • Siddharth Kulkarni
    • 1
  1. 1.Intel CorporationHillsboroUSA

Personalised recommendations