Skip to main content
SpringerLink
Log in

On-the-fly adaptive routing for dragonfly interconnection networks

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Adaptive deadlock-free routing mechanisms are required to handle variable traffic patterns in dragonfly networks. However, distance-based deadlock avoidance mechanisms typically employed in Dragonflies increase the router cost and complexity as a function of the maximum allowed path length. This paper presents on-the-fly adaptive routing (OFAR), a routing/flow-control scheme that decouples the routing and the deadlock avoidance mechanisms. OFAR allows for in-transit adaptive routing with local and global misrouting, without imposing dependencies between virtual channels, and relying on a deadlock-free escape subnetwork to avoid deadlock. This model lowers latency, increases throughput, and adapts faster to transient traffic than previously proposed mechanisms. The low capacity of the escape subnetwork makes it prone to congestion. A simple congestion management mechanism based on injection restriction is considered to avoid such issues. Finally, reliability is considered by introducing mechanisms to find multiple edge-disjoint Hamiltonian rings embedded on the dragonfly, allowing to use multiple escape subnetworks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. We do not consider the opposite case (switching to minimal after a first nonminimal local hop which corresponds to the global misrouting) because we model the MM+L global link selection policy [9] which does not make a first local hop for global misrouting at injection. However, since OFAR decouples the router resources and the path length, it would also support that case.

  2. A minimum occupancy might be also required in the minimal queue to allow for misrouting; we have not considered such a threshold in this work.

  3. We do not show results of ADVG+1 as it could be argued that the additional ring link between the source and destination groups favors the OFAR models. However, since the escape network utilization is only used to avoid potential deadlock situations and not to carry traffic to the destination, the results are similar.

References

  1. Arimilli B, Arimilli R, Chung V, Clark S, Denzel W, Drerup B, Hoefler T, Joyner J, Lewis J, Li J et al (2010) The PERCS high-performance interconnect. In: 2010 18th IEEE symposium on high performance interconnects. IEEE, pp 75–82

  2. Bhatele A, Gropp WD, Jain N, Kale LV (2011) Avoiding hot-spots on two-level direct networks. In: 2011 international conference for high performance computing, networking, storage and analysis (SC), pp 1–11

  3. Brookes S, Roscoe A (1991) Deadlock analysis in networks of communicating processes. Distrib Comput 4(4):209–230

    Article  MATH  MathSciNet  Google Scholar 

  4. Carrion C, Beivide R, Gregorio J, Vallejo F (1997) A flow control mechanism to avoid message deadlock in k-ary n-cube networks. In: Proceedings of fourth international conference on high-performance computing, 1997, pp 322–329. doi:10.1109/HIPC.1997.634510

  5. Cidon I, Ofek Y (1993) Metaring-a full-duplex ring with fairness and spatial reuse. IEEE Trans Commun 41(1):110–120. doi:10.1109/26.212370

    Article  Google Scholar 

  6. Duato J (1995) A necessary and sufficient condition for deadlock-free adaptive routing in wormhole networks. IEEE Trans Parallel Distrib Syst 6(10):1055–1067

    Article  Google Scholar 

  7. Faanes G, Bataineh A, Roweth D, Court T, Froese E, Alverson B, Johnson T, Kopnick J, Higgins M, Reinhard J (2012) Cray cascade: a scalable HPC system based on a dragonfly network. In: International conference on high performance computing, networking, storage and analysis, SC ’12. IEEE Computer Society Press, Los Alamitos, pp 103:1–103:9

  8. García M, Fuentes P, Odriozola M, Vallejo E, Beivide R (2014) FOGSim interconnection network simulator. University of Cantabria. https://code.google.com/p/fogsim/

  9. García M, Vallejo E, Beivide R, Odriozola M, Camarero C, Valero M, Labarta J, Rodríguez G (2013) Global misrouting policies in two-level hierarchical networks. In: Interconnection network architecture: on-chip, multi-chip, pp 13–16

  10. García M, Vallejo E, Beivide R, Odriozola M, Camarero C, Valero M, Rodríguez G, Labarta J, Minkenberg C (2012) On-the-fly adaptive routing in high-radix hierarchical networks. In: International conference on parallel processing (ICPP)

  11. García M, Vallejo E, Beivide R, Valero M, Rodríguez G (2013) OFAR-CM: efficient dragonfly networks with simple congestion management. In: 2013 IEEE 21st annual symposium on high-performance interconnects (HOTI), pp 55–62. doi:10.1109/HOTI.2013.16

  12. Garcia PJ (2011) Congestion management in HPC interconnection networks. HPC Advisory Council European Workshop

  13. Gunther K (1981) Prevention of deadlocks in packet-switched data transport systems. IEEE Trans Commun 29(4):512–524. doi:10.1109/TCOM.1981.1095021

    Article  Google Scholar 

  14. Gupta P, McKeown N (1999) Designing and implementing a fast crossbar scheduler. Micro IEEE 19(1):20–28

    Article  Google Scholar 

  15. IEEE 802 LAN/MAN Standards Committee (2004) IEEE 802.1d-2004 MAC bridges

  16. IEEE 802 LAN/MAN Standards Committee (2010) IEEE standard for local and metropolitan area networks–virtual bridged local area networks–amendment: 10: Congestion notification, 802.1Qau

  17. Jacobson V (1988) Congestion avoidance and control. ACM SIGCOMM Comput Commun Rev 18:314–329

    Article  Google Scholar 

  18. Jiang N, Kim J, Dally WJ (2009) Indirect adaptive routing on large scale interconnection networks. In: ISCA ’09: 36th international symposium on computer architecture

  19. Kerbyson DJ, Barker KJ (2011) Analyzing the performance bottlenecks of the POWER7-IH network. In: CLUSTER. IEEE, pp 244–252

  20. Kermani P, Kleinrock L (1976) Virtual cut-through: a new computer communication switching technique. Comput Netw 3(4):267–286

    MathSciNet  Google Scholar 

  21. Kim J, Dally W, Scott S, Abts D (2008) Technology-driven, highly-scalable dragonfly topology. In: Proceedings of the 35th annual international symposium on computer architecture. IEEE Computer Society, pp 77–88

  22. Lam S, Reiser M (1979) Congestion control of store-and-forward networks by input buffer limits—an analysis. IEEE Trans Commun 27(1):127–134. doi:10.1109/TCOM.1979.1094280

    Article  Google Scholar 

  23. Pinkston T (2004) Deadlock characterization and resolution in interconnection networks. In: Deadlock resolution in computer-integrated systems, CRC Press, pp 445–492

  24. Prisacari B, Rodriguez G, Garcia M, Vallejo E, Beivide R, Minkenberg C (2014) Performance implications of remote-only load balancing under adversarial traffic in dragonflies. In: 8th international workshop on interconnection network architecture: on-chip, multi-chip, INA-OCMC ’14. doi: 10.1145/2556857.2556860

  25. Silla F, Duato J (2000) High-performance routing in networks of workstations with irregular topology. IEEE Trans Parallel Distrib Syst 11(7):699–719. doi:10.1109/71.877816

  26. Valiant L (1982) A scheme for fast parallel communication. SIAM J Comput 11:350

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work has been supported by the Spanish Ministry of Education, FPU grant AP2010-4900; the Spanish Science and Technology Commission (CICYT) under contracts TIN2010-21291-C02-02, TIN2012-34557 and TIN2013- 46957-C2-2-P; the European Union FP7 under Agreements ICT-288777 (Mont-Blanc) and ERC-321253 (RoMoL); the European HiPEAC Network of Excellence and the JSA no. 2013-119 as part of the IBM/BSC Technology Center for Supercomputing agreement.

Author information

Authors and Affiliations

  1. University of Cantabria, Santander, Spain

    Marina García, Enrique Vallejo, Ramón Beivide & Cristóbal Camarero

  2. Barcelona Supercomputing Center and Universitat Politècnica de Catalunya, Barcelona, Spain

    Mateo Valero

  3. IBM Research - Zurich, Rueschlikon, Switzerland

    Germán Rodríguez & Cyriel Minkenberg

Authors
  1. Marina García
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Enrique Vallejo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ramón Beivide
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cristóbal Camarero
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mateo Valero
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Germán Rodríguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Cyriel Minkenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Enrique Vallejo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

García, M., Vallejo, E., Beivide, R. et al. On-the-fly adaptive routing for dragonfly interconnection networks. J Supercomput 71, 1116–1142 (2015). https://doi.org/10.1007/s11227-014-1357-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-014-1357-9

Keywords

Search

Navigation