Lightweight Messages: True Zero-Copy Communication for Commodity Gigabit Ethernet

  • Hai Jin
  • Minghu Zhang
  • Pengliu Tan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4097)


Gigabit Ethernet has become the main cluster interconnection for its low price and well backward compatibility. But the communication performance through Gigabit Ethernet is quite disappointing due to its performance discre- pancies between the hardware and the communication software. To achieve over two-third physical bandwidth of a gigabit network interface, zero-copy protocol architecture is absolutely necessary. Unfortunately, it is impossible to realize true zero-copy communication over non re-programmable commodity Gigabit Ethernet adapters because the DMA engines cannot separate the protocol headers from the payload data directly. This paper introduces LM (Lightweight Messages), a true zero-copy communication mechanism which combines the application level fragmentation scheme with the driver level defragmentation scheme, for existing non-programmable Gigabit Ethernet adapters. Finally, experimental results show that LM can provide better performance than other lightweight communication approaches over commodity Gigabit Ethernet.


Admission Control Payload Data Ethernet Frame Protocol Header Socket Buffer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Pratt, I., Fraser, K.: Arsenic: a user-accessible gigabit Ethernet interface. In: Proceedings of the 12th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM 2001), April 22-26, vol. 1, pp. 67–76 (2001)Google Scholar
  2. 2.
    NetPIPE: A Network Protocol Independent Performance Evaluator, Available:
  3. 3.
    Chu, H.K.J.: Zero-Copy TCP in Solaris. In: Proceedings of the USENIX 1996 Annual Technical Conference, San Diego, CA, USA, January 1996, pp. 253–264 (1996)Google Scholar
  4. 4.
    Balaji, P., Shivam, P., Wyckoff, P., Panda, D.: High Performance User Level Sockets over Gigabit Ethernet. In: Proceeding of IEEE International Conference on Cluster Computing, September 2002, pp. 179–186 (2002)Google Scholar
  5. 5.
    Ciaccio, G., Ehlert, M., Schnor, B.: Exploiting Gigabit Ethernet capacity for cluster applications. In: Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN 2002), November 6-8, pp. 669–678 (2002)Google Scholar
  6. 6.
    Kurmann, C., Muller, M., Rauch, F., Stricker, T.: Speculative Defragmentation - a Technique to Improve the Communication Software Efficiency for Gigabit Ethernet. In: Proceedings of the 9th International Symposium on High-Performance Distributed Computing, Pittsburgh, PA, USA, pp. 131–138 (2000)Google Scholar
  7. 7.
    Eiken, V., Basu, A., Buch, V., Vogels, W.: U-net: A userlevel network interface for parallel and distributed computing. In: Proceedings of the 15th ACM Symposium on Operating Systems Principles (SOSP-15), Cooper Mountain, CO, USA (December 1995)Google Scholar
  8. 8.
    Druschel, P., Peterson, L.L.: FBufs: A high-bandwidth crossdomain transfer facility. In: Proceedings of the 4th ACM Symposium on Operating System Principles, Asheville, NC, December 1993, pp. 189–202 (1993)Google Scholar
  9. 9.
    Dunning, D., Regnier, G., McAlpine, G., Cameron, D., Shubert, B., Berry, F., Merritt, A., Gronke, E., Dodd, C.: The virtual interface architecture. IEEE Micro. 18(2), 66–76 (1998)CrossRefGoogle Scholar
  10. 10.
    Pai, V.S., Druschel, P., Zwaenepoel, W.: I/O-Lite: A unified I/O buffering and caching system. In: Proceedings of the Third Symposium on Operating Systems Design and Implementation (OSDI 1999), pp. 15–28 (1999)Google Scholar
  11. 11.
    Carter, J.B., Zwaenepoel, W.: Optimistic Implementation of Bulk Data Transfer Protocols. In: Proceedings of the 1989 Sigmetrics Conference, May 1989, pp. 61–69 (1989)Google Scholar
  12. 12.
    O’Malley, S.W., Abbot, M.B., Hutchinson, N.C., Peterson, L.L.: A transparent blast facility. Internetworking: Research and Experience 1(2) (December 1990)Google Scholar
  13. 13.
    Tezuka, H., Hori, A., Ishikawa, Y., Sato, M.: PM: An Operating System Coordinated High Performance Communication Library. In: Proceedings of High-Performance Computing and Networking (1997)Google Scholar
  14. 14.
    Shivam, P., Wyckoff, P., Panda, D.: EMP: Zero-copy OSbypass NIC-driven Gigabit Ethernet Message Passing. In: Proceedings of 2001 International Conference on supercomputing (SC 2001), Denver, Colorado, USA (November 2001)Google Scholar
  15. 15.
    Diaz, A.F., Ortega, J., Canas, A., Fernandez, F.J., Anguita, M., Prieto, A.: The lightweight protocol CLIC on Gigabit Ethernet. In: Proceedings of International Parallel and Distributed Processing Symposium (IPDPS 2003) (April 2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Hai Jin
    • 1
  • Minghu Zhang
    • 1
  • Pengliu Tan
    • 1
  1. 1.Cluster and Grid Computing LabSchool of Computer Science and Technology, Huazhong University of Science and TechnologyWuhanChina

Personalised recommendations