Abstract
The communication architecture of the DIMMnet-1 network interface based on MEMOnet is described. MEMOnet is a class of a network interface plugged into a memory slot. This paper proposes three message transfer mechanisms named atomic on-the-fly sending (AOTF), block on-the-fly sending (BOTF) and OTF receiving with selective address translation. The DIMMnet-1 prototype will have an ASIC named Martini, two banks of PC133 based SO-DIMM slots and an 8 Gbps full duplex optical link. The software overhead incurred to generate a message is only 1 CPU cycle and the estimated hardware delay is 105 ns using AOTF. The estimated hardware delay for receiving to on chip memory using OTF receiver is 90 ns. The estimated achievable sending bandwidth of DIMMnet-1 using BOTF is 984 MB/s which was observed in our experiments. This bandwidth is 7.4 times higher than the maximum bandwidth of PCI. This high performance is available even when simultaneous sending and receiving are executed on a cheap personal computer with DIMM slots. This paper also discribes the effects of BOTF for a PCI-based NIC.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Myricom Corp., http://www.myri.com/.
Dolphin Corp. PCI SCI-64 Adapter Card, http://www.dolphinics.no/ dics_pci-sci64.htm.
InfiniBand Trade Association, http://www.infinibandta.org/.
T. Kudoh, S. Nishimura, J. Yamamoto, H. Nishi, O. Tatebe and H. Amano, RHiNET: A network for high performance parallel processing using locally distributed computers, IWIA'99 (November 1999).
Yamamoto, Tatebe, Yokoyama, Tsuchiya, Miyawaki, Shimizu, Amano and Kudoh, Implementation and evaluation of RHiNET-1, a network for LASN, in: IPSJ SIG Notes 99-ARC-137 (HOKKE 2000) (March 2000) pp. 59–64 (in Japanese).
S. Nishimura, T. Kudoh, H. Nishi, J. Yamamoto, K. Harasawa, N. Matsudaira, S. Akutsu, K. Tasho and H. Amano, High-speed network switch RHiNET-2/SW and its implementation with optical interconnections, in: Symposium on High Performance Interconnects (HOT INTERCONNECTS 8) (August 2000).
R. Ueno, S. Inasawa, H. Nishi, T. Kudoh and H. Amano, Flow control method in high speed transfer using optical interconnect, in: IPSJ SIG Notes 00-ARC-145 (December 2000) (in Japanese).
T. Yoshikawa and H. Matsuoka, Optical interconnections for parallel and distributed computing, Proceedings of the IEEE 88(6) (June 2000) 849–855.
N. Tanabe, J. Yamamoto and T. Kudoh, MEMnet: Network interface attached on memory slot, in: IPSJ SIG Notes 99-ARC-134 (SWoPP'99) (August 1999) pp. 73–78 (in Japanese).
N. Tanabe, J. Yamamoto, H. Nishi, T. Kudoh, Y. Hamada, H. Nakajo and H. Amano, MEMOnet: Network interface plugged into a memory slot, in: IEEE Int. Conference on Cluster Computing (CLUSTER2000) (November 2000) pp. 17–26.
N. Tanabe, Multi Processor System, in: Japan Patent Application No. PH02157491 (June 1990), Publication No. PH04048371 (February 1992) (in Japanese).
R. Minnich, D. Burns and F. Hady, The memory integrated network interface, IEEE Micro 15 (February 1995).
N. Tanabe et al., Base-m n-cube: High performance interconnection networks for highly parallel computer PRODIGY, in: Proc. of 1991 Int. Conference on Parallel Procesing (20th ICPP) (August 1991) pp. I509–516.
Intel Corp. AGP Inline Memory Module Specificationi, available from http://developer.intel.com/technology/memory/aimm/ 29817704.pdf.
Standard of Electronic Industries Association of Japan, Processor Enhanced Memory Module (PEMM) Standard for Processor Enhanced Memory Module Functional Specifications, EIAJ ED-5514 (July 1998).
Loom Shuttle com., PEMM Information and Links, http://www. loomshuttle.com/PEMM/PEMMinfo.html.
Compac Corp. PCI-X Industry Enablement Program, http://www. compaq.com/products/servers/technology/pci-x-enablement.html.
Y. Ishikawa, A. Hori, H. Tezuka, S. Sumimoto, T. Takahashi and H. Harada, RWCP PC Cluster Programming Environment — Extended Abstract, in: Extreme Linux Workshop #2, USENIX (1999) pp. 16–19.
D.J. Scales, K. Gharachorloo and C.A. Thekkath, Shasta: A low overhead, software-only approach for supporting fine-grain shared memory, ASPLOS'96 (October 1996).
M. Fillo and R.B. Gillett, Architecture and implementation of MEMORY CHANNEL 2, Digital Technical Journal 9(1) (1997).
Giganet corp. cLAN for Linux Data Sheet, available from http://www. giganet.com/pdf/linuxadapter.pdf.
H. Tezuka, A. Hori, Y. Ishikawa and M. Sato, PM: An operating system coordinated high performance communication library, in: High Performance Computing and Networking'97 (1997).
S. Sumimoto, H. Tezuka, A. Hori, H. Harada, T. Takahashi and Y. Ishikawa, The design and evaluation of high performance communication using a Gigabit Ethernet, in: Int. Conference on Supercomputing' 99, ACM SIGARCH (June 1999) pp. 243–250.
S. Sumimoto, A. Hori, H. Tezuka, H. Harada, T. Takahashi and Y. Ishikawa, GigaE PM II: Design of high performance comminication library using Gigabit Ethernet, in: IPSJ SIG Notes 99-ARC-134 (SWoPP'99) (August 1999) pp. 61–66 (In Japanese).
Intel Corp. Block Copy Using Pentium III Streaming SIMD Extensions Revision 1.9, available from ftp://download.intel.com/design/servers/ softdev/copy.pdf (January 1999).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Tanabe, N., Yamamoto, J., Nishi, H. et al. Low Latency High Bandwidth Message Transfer Mechanisms for a Network Interface Plugged into a Memory Slot. Cluster Computing 5, 7–17 (2002). https://doi.org/10.1023/A:1012732403321
Issue Date:
DOI: https://doi.org/10.1023/A:1012732403321