The Journal of Supercomputing

, Volume 27, Issue 2, pp 103–128 | Cite as

A Scalable Interconnection Network Architecture for Petaflops Computing

  • Constantine Katsinis
  • Bahram Nabet


Extrapolating technology advances in the near future, a computer architecture capable of petaflops performance will likely be based on a collection of processing nodes interconnected by a high-performance network. One possible organization would consist of thousands of inexpensive, low-power symmetric multiprocessor (SMP) nodes. Each node will inject data into the interconnection network at a very large rate and consequently, the interconnect scheme is one of the most crucial design issues affecting system performance. This paper describes the 2D simultaneous optical multiprocessor exchange bus (2D SOME-Bus) which has the potential to become the basis of a high-end computer architecture capable of petaflops performance. It consists of N horizontal, N vertical 1D SOME-Bus networks, and N2 nodes. Each node is connected to one horizontal and one vertical 1D SOME-Bus. Each of N nodes connected to one 1D SOME-Bus has a dedicated broadcast channel and an input channel interface based on an array of N receivers monitoring all N channels and allowing multiple simultaneous broadcasts. In the 2D SOME-Bus, messages being broadcast on one Bus can be broadcast in a cut-through manner on one or more Buses in the other dimension. This paper describes the optoelectronic devices and technology which make the 2D SOME-Bus possible, and the network interface organization. It also presents simulation results which compare the performance of the 2D SOME-Bus, the 1D SOME-Bus, the crossbar and the torus under the message-passing paradigm.

petaflops computing computer architecture interconnection networks performance analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    A. Anwar and B. Nabet. Barrier enhancement mechanisms in heterodimensional contacts and their effect on current transport. IEEE Trans. on Microwave Theory and Techniques, 50(1):68–71, pt. 1, January 2002.Google Scholar
  2. 2.
    A. Anwar, B. Nabet, and F. Castro. Effects of electron confinement on thermionic emission current in a modulation doped heterostructure. Journal of Applied Physics, 85(5):2663–2666, March 1999.Google Scholar
  3. 3.
    S. Attaway. Transient solid dynamics simulations on the Sandia/Intel teraflop computer. Supercomputing 97, Scholar
  4. 4.
    A. Bouzid and M. A. G. Abushagur. Thin-film approximate modeling of in-core fiber gratings. Optical Engineering, 35(10):2793–2797, 1996.Google Scholar
  5. 5.
    P. Bunyk. RSFQ Subsystem for petaflops-scale computing: COOL-0. Proc. Third Petaflop Workshop, pp. 3–9, 1999.Google Scholar
  6. 6.
    D. A. Carlson. Modified-mesh connected parallel computers. IEEE Trans. on Computers, 37(10):1315–1321, October 1988.Google Scholar
  7. 7.
    K. L. Chung. Prefix computations on a generalized mesh-connected computer with multiple buses. IEEE Trans. Parallel and Distributed Systems, 6(2):196–199, 1995.Google Scholar
  8. 8.
    J. Culp, B. Nabet, F. Castro, and A. Anwar. Gain enhancement of low temperature GaAs heterojunction MSM photodtector. Lasers and Electro Optics Conference, 6:274–275, 1998.Google Scholar
  9. 9.
    D. Delagebeudeuf and N. T. Linh. Metal-(n) AlGaAs-GaAs two-dimensional electron gas FET. IEEE Trans. Elec. Dev., 39(5):955–960, 1982.Google Scholar
  10. 10.
    J. Demmel. Performance of a parallel global atmospheric chemical tracer model. Supercomputing 96, www.supercomp.orgGoogle Scholar
  11. 11.
    O. M. Dighe, R. Vaidyanathan, and S. Q. Zheng. The bus-connected ringed tree: A versatile interconnection network. J. Parallel and Distributed Computing, 33:189–196, 1996.Google Scholar
  12. 12.
    L. Dong, B. Ortega, and L. Reekie. Coupling characteristics of claddding modes in tilted optical fiber gratings. Applied Optics, 37(22):5099–5105, 1998.Google Scholar
  13. 13.
    J. J. Dongarra and D. W. Walker. The quest for petascale computing. Computing in Science & Engineering, 32–39, May/June 2001.Google Scholar
  14. 14.
    T. Erdogan and J. Sipe. Tilted fiber phase gratings. Journal of the Optical Society of America, 13(2):296–313, 1996.Google Scholar
  15. 15.
    C. Evangelinos. Communication performance models in prism. A spectral element-Fourier parallel Navier-stokes solver. Supercomputing 95, www.supercomp.orgGoogle Scholar
  16. 16.
    G. Fox and W. Furmanski. Petaops and exaops: Supercomputing on the Web. IEEE Internet Computing, 38–42, March–April 1997.Google Scholar
  17. 17.
    G. Gao, K. Likharev, P. Messina, and T. Sterling. Hybrid technology multithreaded architecture. IEEE Frontiers of Massively Parallel Computation Conf, 98–105, 1996.Google Scholar
  18. 18.
    B. Gelmont, M. Shur, and C. Moglestue. Theory of junctions between two-dimensional electron gas and p-type semiconductor. IEEE Trans. Elec. Dev., 39(5):1216–1222, 1992.Google Scholar
  19. 19.
    J. R. Goodman and P. J. Woest. The Wisconsin Multicube: A new large scale cache coherent multiprocessor. 15th International Symposium on Computer Architecture, 422–431, 1988.Google Scholar
  20. 20.
    G. Gravenstreter and R. Melhem. Realizing common communication patterns in partitioned optical passive stars (POPS) networks. IEEE Trans. on Computers, 47(9):998–1013, 1998.Google Scholar
  21. 21.
    S. Gupta, M. Y. Frankel, J. A. Valdmanis, J. F. Whitaker, G. A. Mourou, F. W. Smith, and A. R. Calawa. Appl. Phys. Lett., 59:3276–3277, 1991.Google Scholar
  22. 22.
    B. Hendrickson. Parallel many-body simulations without all-to-all communication. Journal Parallel Distributed Computing, 27(1):15–25, 1995.Google Scholar
  23. 23.
    M. Hortsmann, M. Marso, A. Fox, F. Ruders, H. Hollfelder, H. Hardtdegen, P. Kordos, and H. Luth. InP/InGaAs photodetector based on a high electron mobility transistor layer structure: Its response at 1.3 µm wavelength. Applied Physics Letters, 67(1):106–108, July 3, 1995.Google Scholar
  24. 24.
    M. Kaminska, Z. Liliental-Weber, E. R. Weber, T. George, J. B. Cortright, F. W. Smith, B. Y. Tsaur, and A. R. Calawa. Appl. Phys. Lett., 54:1881–1882, 1989.Google Scholar
  25. 25.
    C. Katsinis, W. Cohen, R. Gaede, and J. Kulick. The architecture and performance of the simultaneous optical multiprocessor exchange interconnection network (SOME-Bus). International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'97), Las Vegas, NV, June 1997.Google Scholar
  26. 26.
    C. Katsinis. Performance analysis and simulation of the SOME-Bus architecture using message passing. 7th International Conference on Computer Communications and Networks (ICCCN98), pp. 68–72, Lafayette, LA, October 1998.Google Scholar
  27. 27.
    C. Katsinis. Performance analysis of the simultaneous optical multiprocessor exchange bus. Parallel-Computing, 27(8):1079–1115, July 2001.Google Scholar
  28. 28.
    P. M. Kogge. In pursuit of the petaflop: overcoming the latency/bandwidth wall with PIM technology. Second Conference on Enabling Technologies for Petaflops Computing, February 1999.Google Scholar
  29. 29.
    M. Lee and G. Little. Study of radiation modes for 45-deg tilted fiber phase gratings. Optical Engineering, 37(10):2687–2698, 1998.Google Scholar
  30. 30.
    Y. Li. Optical multiple-access mesh-connected bus interconnections. IEEE Proceedings 82(11):1690, November 1994.Google Scholar
  31. 31.
    D. C. Look, D. C. Walters, M. Mier, C. E. Stutz, and S. K. Brierley. Native donors and acceptors in molecular-beam epitaxial GaAs grown at 200°C. Applied Physics Letters, 60(23):2900–2902, June 8, 1992.Google Scholar
  32. 32.
    D. C. Look, G. D. Robinson, J. R. Sizelove, and C. E. Stutz. Donor and acceptor concentrations in molecular beam epitaxial GaAs grown at 300 and 400°C. Applied Physics Letters, 62(23):3004–3006, June 7, 1993.Google Scholar
  33. 33.
    J. Lou. Performance analysis and optimization on the UCLA parallel atmospheric general circulation model code. Supercomputing 95, www.supercomp.orgGoogle Scholar
  34. 34.
    A. Louri and B. Weech. A spanning multichannel linked hypercube: A gradually scalable optical interconnection network for massively parallel computing. IEEE Trans. on Parallel and Distributed Systems, 9(5):497–512, 5/1998.Google Scholar
  35. 35.
    P. Messina, D. Culler, W. Pfeiffer, W. Martin, J. Oden, and G. Smith. Architecture. Comm. ACM, 41(11):36–44, November 1998.Google Scholar
  36. 36.
    A. S. Morris. In search of transparent networks. IEEE Spectrum, 47–51, October 2001.Google Scholar
  37. 37.
    B. Nabet, A. Paolella, P. Cooke, M. Lemuene, R. P. Moerkirk, and L.-C. Liou. Effect of MBE growth temperature on large area MSM detectors. Applied Physics Letters, 64(23):3151–3153, 1994.Google Scholar
  38. 38.
    B. Nabet. A heterojunction metal-semiconductor-metal photodetector. IEEE Photonics Tech. Lett., 9(2):223–225, 1997.Google Scholar
  39. 39.
    B. Nabet, F. Castro, A. Anwar, and A. Cola. Heterodimensional contacts and optical detection (invited). International Journal of High Speed Electronics and Systems, 10(1):375–386, March 2000.Google Scholar
  40. 40.
    B. Nabet, et al. Electron cloud effect on current injection across a Schottky contact. Applied Physics Let., 4007–4010, December 11, 2000.Google Scholar
  41. 41.
    A. Nowatzyk. Are crossbars really dead? The case for optical multiprocessor interconnection systems. Computing Architecture News, 23(2):106, May 1995.Google Scholar
  42. 42.
    M. Ould-Khaoua. Comparative evaluation of hypermesh and multi-stage interconnection network. Computer Journal, 39(3):232, 1996.Google Scholar
  43. 43.
    H. F. B. Ozelo, L. E. M. de Barros Jr., B. Nabet, L. G. Neto, M. A. Romero, and J. W. Swart. MSM photodetector with an integrated microlens array for improved optical coupling. Int. Microwave and Optoelectronics Conference (IMOC'99), pp. 472–475, Rio de Janeiro, Brazil, August 1999.Google Scholar
  44. 44.
    Y. Pan, S. Q. Zheng, Keqin Li, and Hong Shen. An improved generalization of mesh-connected computers with multiple buses. IEEE Trans. on Parallel and Distributed Systems, 12(3):293–305, March 2001.Google Scholar
  45. 45.
    S. Plimpton. Transient dynamics simulations. Parallel algorithms for contact detection and smoothed particle hydrodynamics. Supercomputing 96, www.supercomp.orgGoogle Scholar
  46. 46.
    C. Qiao and R. Melhem. Reducing communication latency with path multiplexing in optically interconnected multiprocessor systems. IEEE Trans. on Parallel and Distributed Systems, 8(2):97–108, 1997.Google Scholar
  47. 47.
    S. Rajasekaran and S. Sahni. Sorting, selection, and routing on the array with reconfigurable optical buses. IEEE Trans. on Parallel and Distributed Systems, 8(11):1123–1132, 1997.Google Scholar
  48. 48.
    International Technology Roadmap for Semiconductors, Semiconductor Industry Association, Austin, Texas, 1999.Google Scholar
  49. 49.
    F. W. Smith, A. R. Calawa, C. L. Chen, M. J. Manfra, and L. J. Mahoney. New MBE buffer used to eliminate backgating in GaAs MESFETs. IEEE Electron Device Lett. 9(77):77–80, 1988.Google Scholar
  50. 50.
    T. Sterling, P. Messina, and P. Smith. Enabling Technology for Petaflops Computers, MIT Press, Cambridge, MA, 1995.Google Scholar
  51. 51.
    R. Stevens. The August. 1995 petaflops workshop. Scholar
  52. 52.
    T. Szymanski. Hypermeshes. optical interconnection network for parallel computing. Journal Parallel Distributed Computing, 26(1), April 1995.Google Scholar
  53. 53.
    B. C. Tousley, N. Davids, A. H. Sayles, A. Paolella, P. Cooke, M. L. Lemuene, and R. P. Moerkirk, and B. Nabet. Broad-bandwidth, high-responsivity intermediate growth temperature GaAs MSM photodetectors. Photonics Technology Letters, 7(12):1483–1485, December 95.Google Scholar
  54. 54.
    Tsai, Horng-Ren, Solving an algebraic path problem and some related graph problems on a hyper-bus broadcast network. IEEE Trans. on Parallel and Distributed Systems, 8(12):1226–1235, December 1997.Google Scholar
  55. 55.
    S. Wallach. Petaflop architectures. Second Conference on Enabling Technologies for Petaflops Computing, February 1999.Google Scholar
  56. 56.
    M. Warren. Parallel supercomputing with commodity components. International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'97), pp. 1372–1376, Las Vegas, Nevada, June 1997.Google Scholar
  57. 57. Scholar
  58. 58.
    B. J. V. Zeghbroeck, W. Patrick, J. M. Halbout, and P. Vettiger. 105-Ghz bandwidth metalsemiconductor-metal photodiode. IEEE Elec. Dev. Lett., 9(10):527–529, October 1988.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Constantine Katsinis
  • Bahram Nabet

There are no affiliations available

Personalised recommendations