Cluster Computing

, Volume 6, Issue 2, pp 125–142 | Cite as

Performance Evaluation of the Quadrics Interconnection Network

  • Fabrizio Petrini
  • Eitan Frachtenberg
  • Adolfy Hoisie
  • Salvador Coll


In this paper we present an in-depth description of the Quadrics interconnection network (QsNET) and an experimental performance evaluation on a 64-node AlphaServer cluster. We explore several performance dimensions and scaling properties of the network by using a collection of benchmarks, based on different traffic patterns. Experiments with permutation patterns and uniform traffic are conducted to illustrate the basic characteristics of the interconnect under conditions commonly created by parallel scientific applications. Moreover, the behavior of the QsNET under I/O traffic, and the influence of the placement of the I/O servers are analyzed. The effects of using dedicated I/O nodes or shared I/O nodes are also exposed. In addition, we evaluate how background I/O traffic interferes with other parallel applications running concurrently. The experimental results indicate that the QsNET provides excellent performance in most cases, with excellent contention resolution mechanisms. Some important guidelines for applications and I/O servers mapping on large-scale clusters are also given.

interconnection networks performance evaluation user-level communication operating system bypass 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    N.J. Boden, D. Cohen, R.E. Felderman, A.E. Kulawick, C.L. Seitz, J.N. Seizovic and W.-K. Su, Myrinet: A gigabit-per-second local area network, IEEE Micro (1) (1995) 29–36.Google Scholar
  2. [2]
    H. Chen and P. Wyckoff, Simulation studies of Gigabit Ethernet versus Myrinet using real application cores, in: Proceedings of CANPC'00, Workshop of High-Performance Computer Architecture, Toulouse, France, January 2000.Google Scholar
  3. [3]
    W.J. Dally and C.L. Seitz, Deadlock-free message routing in multi-processor interconnection networks, IEEE Transactions on Computers C-36(5) (1987) 547–553.Google Scholar
  4. [4]
    J. Duato, S. Yalamanchili and L. Ni, Interconnection Networks: An Engineering Approach (IEEE Computer Society Press, 1997).Google Scholar
  5. [5]
    A. Griest, W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, W. Saphir, T. Skjellum and M. Snir, MPI-2: Extending the message passing interface, in: 2nd International Euro-Par Conference, Lyon, France, Vol. 1, Lecture Notes in Computer Science, Vol. 1123 (Springer, 1996) pp. 128–135.Google Scholar
  6. [6]
    S. Heller, Congestion-free routing on the CM-5 data router, in: 1st International Workshop, PCRCW'94, Seattle, Washington, USA, Lecture Notes in Computer Science, Vol. 853, eds. K. Bolding and L. Snyder (Springer, 1994) pp. 176–184.Google Scholar
  7. [7]
    H. Hellwagner, The SCI standard and applications of SCI, in: SCI: Scalable Coherent Interface, Lecture Notes in Computer Science, Vol. 1291, eds. H. Hellwagner and A. Reinfeld (Springer, 1999) pp. 95–116.Google Scholar
  8. [8]
    InfiniBand Specification 1.0a. InfiniBand Trade Association (June 2001).Google Scholar
  9. [9]
    D. Kerbyson, H. Alme, A. Hoisie, F. Petrini, H. Wasserman and M. Gittings, Predictive performance and scalability modeling of a largescale application, in: Supercomputing 2001, Denver, CO, November 2001.Google Scholar
  10. [10]
    C.E. Leiserson, Fat-trees: Universal networks for hardware efficient supercomputing, IEEE Transactions on Computers C-34(10) (1985) 892–901.Google Scholar
  11. [11]
    L.M. Ni, Y. Gui and S. Moore, Performance evaluation of switch-based wormhole networks, IEEE Transactions on Parallel and Distributed Systems 8(5) (1997) 462–474.Google Scholar
  12. [12]
    F. Petrini, Communication performance of wormhole interconnection networks, Ph.D. thesis, Università degli Studi di Pisa, Dipartimento di Informatica, February 1997.Google Scholar
  13. [13]
    F. Petrini, W. Chun Feng, A. Hoisie, S. Coll and E. Frachtenberg, The quadrics network: High performance clustering technology, IEEE Micro 22(1) (January–February 2002) 46–57.Google Scholar
  14. [14]
    F. Petrini, S. Coll, E. Frachtenberg and A. Hoisie, Hardware-and software-based collective communication on the Quadrics network, in: IEEE International Symposium on Network Computing and Applications 2001 (NCA 2001), Boston, MA, October 2001.Google Scholar
  15. [15]
    F. Petrini, A. Hoisie, W. Chun Feng and R. Graham, Performance evaluation of the quadrics interconnection network, in: Workshop on Communication Architecture for Clusters (CAC'01), San Francisco, CA, April 2001.Google Scholar
  16. [16]
    F. Petrini and M. Vanneschi, k-ary n-trees: High performance networks for massively parallel architectures, in: Proceedings of the 11th International Parallel Processing Symposium, IPPS'97, Geneva, Switzerland, April 1997, pp. 87–93.Google Scholar
  17. [17]
    F. Petrini and M. Vanneschi, Performance analysis of wormhole routed k-ary n-trees, International Journal on Foundations of Computer Science 9(2) (June 1998) 157–177.Google Scholar
  18. [18]
    G.F. Pfister and V.A. Norton, Hot-spot contention and combining in multistage interconnection networks, IEEE Transactions on Computers C-34(10) (1985) 943–948.Google Scholar
  19. [19]
    Quadrics Supercomputers World Ltd., Elan Programming Manual (January 1999).Google Scholar
  20. [20]
    Quadrics Supercomputers World Ltd., Elan Reference Manual (January 1999).Google Scholar
  21. [21]
    Quadrics Supercomputers World Ltd., Elite Reference Manual (November 1999).Google Scholar
  22. [22]
    R. Seifert, Gigabit Ethernet: Technology and Applications for High Speed LANs (Addison-Wesley, 1998).Google Scholar
  23. [23]
    M. Snir, S. Otto, S. Huss-Lederman, D. Walker and J. Dongarra, MPI–The Complete Reference, The MPI Core, Vol. 1 (TheMIT Press, 1998).Google Scholar
  24. [24]
    D. Tolmie, T.M. Boorman, A. DuBois, D. DuBois, W. Feng and I. Philp, From HiPPI-800 to HiPPI-6400: A changing of the guard and gateway to the future, in: Proceedings of the 6th International Conference on Parallel Interconnects (PI'99), October 1999.Google Scholar
  25. [25]
    W. Vogels, D. Follett, J. Hsieh, D. Lifka and D. Stern, Tree-saturation control in the AC3 velocity cluster, in: Hot Interconnects 8 (Stanford University, Palo Alto CA, 2000).Google Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • Fabrizio Petrini
    • 1
  • Eitan Frachtenberg
    • 1
  • Adolfy Hoisie
    • 1
  • Salvador Coll
    • 1
    • 2
  1. 1.CCS-3 Modeling, Algorithms and InformaticsComputer and Computational Sciences Division, Los Alamos National LaboratoryUSA
  2. 2.Electronic Engineering DepartmentTechnical University of ValenciaSpain

Personalised recommendations