Abstract
Gigabit networks are equipped with “increasingly” intelligent network interface cards, and the firmware running in the cards does various tasks related to end-to-end communication. For an accurate performance evaluation of gigabit networks, it is very important to characterize and quantify the firmware. However, the firmware has been neglected in the latency analyzes of network protocols.
This paper presents an in-depth latency analysis of Myrinet. Our findings include that the major bottleneck is the network interface card itself. This is true especially for so-called lightweight user-level protocols (such as BPI of Myrinet) designed for high-speed communication. Although BPI is very lean and efficient in the host, its sending throughput becomes similar to UDP. This result is very unexpected and surprising. Through firmware-level measurements, we identify that the cause of bottleneck is the DMA performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C. L. Seitz, J. N. Seizovic, and W.-K. Su. Myrinet-A gigabit-per-second local-area network. IEEE-Micro, 15(1):29–36, 1995.
Gigabit Ethernet Alliance. Media Access Control (MAC) Parameters, Physical Layer, Repeater and Management Parameters for 1000 Mbps Operation. IEEE Std 802.3z, 1998.
D. D. Clark, V. Jacobson, J. Romkey, and H. Salwen. An analysis of TCP processing overhead. IEEE Communications Magazine, 27(6):23–29, 1989.
J. Kay and J. Pasquale. Measurement, analysis, and improvement of UDP/IP throughput for the DECstation 5000. Proceedings of USENIX Winter Conference, pp. 249–258, 1993.
A. Wolman, G. Voelker, and C. A. Thekkath. Latency analysis of TCPon an ATM network. Proceedings of USENIX Winter Conference, pp. 167–179, 1994.
C. Yoo, H.-W. Jin, and S.-C. Kwon. Asynchronous UDP. IEICE Transactions on Communications, E84-B(12):3243–3251, 2001.
Myricom Inc. Myrinet User's Guide. http://www.myri.com, 1996.
C. Dalton, G. Watson, D. Banks, C. Calamvokis, A. Edwards, and J. Lumley. Afterburner. IEEE Network, 7(4):36–43, 1993.
P. Druschel and L. L. Peterson. Fbufs: A high-bandwidth cross-domain transfer facility. Proceedings of 14th ACM SOSP, pp. 189–202, 1993.
H. J. Chu. Zero-copy TCPin Solaris. Proceedings of 1996 Winter USENIX, 1996.
J. C. Brustoloni and P. Steenkiste. Copy emulation in checksummed, multiple-packet communication. Proceedings of IEEE Infocom '97, pp. 1124–1132, 1997.
A. Gallatin, J. Chase, and K. Yocum. Trapeze/IP: TCP/IP at near-gigabit speeds. Proceedings of 1999 USENIX Technical Conference, 1999.
H.-W. Jin, C. Yoo, and S.-K. Park. Stepwise optimizations of UDP/IP on a gigabit network. Proceedings of the 8th International Euro-Par Conference (Euro-Par 2002), LNCS, 2400:745–748, 2002.
T. V. Eicken, A. Basu, V. Buch, and W. Vogels. U-Net: A user-level network interface for parallel and distributed computing. Proceedings of 15th ACM SOSP, pp. 40–53, 1995.
S. Pakin, M. Lauria, and A. Chien. High performance messaging on workstations: Illinois fast messages (FM) for Myrinet. Proceedings of Supercomputing '95 (SC95), 1995.
C. Dubnicki, L. Iftode, E. Felten, and K. Li. Software support for virtual memory-mapped communication. Proceedings of the 10th International Parallel Processing Symposium, 1996.
L. Prylli and B. Tourancheau. BIP: A new protocol designed for high performance networking on Myrinet. Proceedings of IPPS/SPDP98, 1998.
D. Dunning, G. Regnier, G. McAlpine, D. Cameron, B. Shubert, A. M. Berry, E. Gronke, and C. Dodd. The virtual interface architecture. IEEE Micro, 18(2):66–76, 1998.
T. Takahashi, S. Sumimoto, A. Hori, H. Harada, and Y. Ishikawa. PM2: A high performance communication middleware for heterogeneous network environments. Proceedings of Supercomputing 2000 (SC2000), 2000.
I. Pratt and K. Fraser. Arsenic: A User-accessible gigabit ethernet interface. Proceedings of IEEE Infocom 2001, pp. 67–76, 2001.
P. Shivam, P. Wyckoff, and D. Panda. EMP: Zero-copy OS-bypass NIC-driven gigabit ethernet message passing. Proceedings of Supercomputing 2001 (SC2001), 2001.
H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson. RTP: A treansport protocol for real-time applications. RFC 1889, 1996.
H.-W. Jin, K.-S. Bang, C. Yoo, J.-Y. Choi, and H.-J. Cha. Bottleneck analysis of a gigabit network interface card: Formal verification approach. Proceedings of the 9th International SPIN Workshop on Model Checking of Software (SPIN 2002), LNCS, 2318:170–186, 2002.
S. Elbert, Q. Snell, A. Mikler, G. Helmer, C. Csanady, K. Stearns, B. MacLeod, M. Johnson, B. Osborn, and I. Verigin. Gigabit ethernet and Low-cost supercomputing. Technical report IS-5126. Ames Laboratory, 1998.
H.-W. Jin and C. Yoo. Latency analysis of UDPand BPI on Myrinet. Proceedings of 18th IEEE International Performance, Computing, and Communication Conference (IPCCC'99), pp. 185–191, 1999.
Intel Co. Pentium Pro Family Developer's Manual, Volume 3: Operating System Writer's Guide. Order Number: 242692, 1995.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Jin, HW., Yoo, C. & Choi, JY. Firmware-Level Latency Analysis on a Gigabit Network. The Journal of Supercomputing 26, 59–75 (2003). https://doi.org/10.1023/A:1024416118111
Issue Date:
DOI: https://doi.org/10.1023/A:1024416118111