Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

On-the-fly adaptive routing for dragonfly interconnection networks

  • 315 Accesses

  • 2 Citations


Adaptive deadlock-free routing mechanisms are required to handle variable traffic patterns in dragonfly networks. However, distance-based deadlock avoidance mechanisms typically employed in Dragonflies increase the router cost and complexity as a function of the maximum allowed path length. This paper presents on-the-fly adaptive routing (OFAR), a routing/flow-control scheme that decouples the routing and the deadlock avoidance mechanisms. OFAR allows for in-transit adaptive routing with local and global misrouting, without imposing dependencies between virtual channels, and relying on a deadlock-free escape subnetwork to avoid deadlock. This model lowers latency, increases throughput, and adapts faster to transient traffic than previously proposed mechanisms. The low capacity of the escape subnetwork makes it prone to congestion. A simple congestion management mechanism based on injection restriction is considered to avoid such issues. Finally, reliability is considered by introducing mechanisms to find multiple edge-disjoint Hamiltonian rings embedded on the dragonfly, allowing to use multiple escape subnetworks.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14


  1. 1.

    We do not consider the opposite case (switching to minimal after a first nonminimal local hop which corresponds to the global misrouting) because we model the MM+L global link selection policy [9] which does not make a first local hop for global misrouting at injection. However, since OFAR decouples the router resources and the path length, it would also support that case.

  2. 2.

    A minimum occupancy might be also required in the minimal queue to allow for misrouting; we have not considered such a threshold in this work.

  3. 3.

    We do not show results of ADVG+1 as it could be argued that the additional ring link between the source and destination groups favors the OFAR models. However, since the escape network utilization is only used to avoid potential deadlock situations and not to carry traffic to the destination, the results are similar.


  1. 1.

    Arimilli B, Arimilli R, Chung V, Clark S, Denzel W, Drerup B, Hoefler T, Joyner J, Lewis J, Li J et al (2010) The PERCS high-performance interconnect. In: 2010 18th IEEE symposium on high performance interconnects. IEEE, pp 75–82

  2. 2.

    Bhatele A, Gropp WD, Jain N, Kale LV (2011) Avoiding hot-spots on two-level direct networks. In: 2011 international conference for high performance computing, networking, storage and analysis (SC), pp 1–11

  3. 3.

    Brookes S, Roscoe A (1991) Deadlock analysis in networks of communicating processes. Distrib Comput 4(4):209–230

  4. 4.

    Carrion C, Beivide R, Gregorio J, Vallejo F (1997) A flow control mechanism to avoid message deadlock in k-ary n-cube networks. In: Proceedings of fourth international conference on high-performance computing, 1997, pp 322–329. doi:10.1109/HIPC.1997.634510

  5. 5.

    Cidon I, Ofek Y (1993) Metaring-a full-duplex ring with fairness and spatial reuse. IEEE Trans Commun 41(1):110–120. doi:10.1109/26.212370

  6. 6.

    Duato J (1995) A necessary and sufficient condition for deadlock-free adaptive routing in wormhole networks. IEEE Trans Parallel Distrib Syst 6(10):1055–1067

  7. 7.

    Faanes G, Bataineh A, Roweth D, Court T, Froese E, Alverson B, Johnson T, Kopnick J, Higgins M, Reinhard J (2012) Cray cascade: a scalable HPC system based on a dragonfly network. In: International conference on high performance computing, networking, storage and analysis, SC ’12. IEEE Computer Society Press, Los Alamitos, pp 103:1–103:9

  8. 8.

    García M, Fuentes P, Odriozola M, Vallejo E, Beivide R (2014) FOGSim interconnection network simulator. University of Cantabria.

  9. 9.

    García M, Vallejo E, Beivide R, Odriozola M, Camarero C, Valero M, Labarta J, Rodríguez G (2013) Global misrouting policies in two-level hierarchical networks. In: Interconnection network architecture: on-chip, multi-chip, pp 13–16

  10. 10.

    García M, Vallejo E, Beivide R, Odriozola M, Camarero C, Valero M, Rodríguez G, Labarta J, Minkenberg C (2012) On-the-fly adaptive routing in high-radix hierarchical networks. In: International conference on parallel processing (ICPP)

  11. 11.

    García M, Vallejo E, Beivide R, Valero M, Rodríguez G (2013) OFAR-CM: efficient dragonfly networks with simple congestion management. In: 2013 IEEE 21st annual symposium on high-performance interconnects (HOTI), pp 55–62. doi:10.1109/HOTI.2013.16

  12. 12.

    Garcia PJ (2011) Congestion management in HPC interconnection networks. HPC Advisory Council European Workshop

  13. 13.

    Gunther K (1981) Prevention of deadlocks in packet-switched data transport systems. IEEE Trans Commun 29(4):512–524. doi:10.1109/TCOM.1981.1095021

  14. 14.

    Gupta P, McKeown N (1999) Designing and implementing a fast crossbar scheduler. Micro IEEE 19(1):20–28

  15. 15.

    IEEE 802 LAN/MAN Standards Committee (2004) IEEE 802.1d-2004 MAC bridges

  16. 16.

    IEEE 802 LAN/MAN Standards Committee (2010) IEEE standard for local and metropolitan area networks–virtual bridged local area networks–amendment: 10: Congestion notification, 802.1Qau

  17. 17.

    Jacobson V (1988) Congestion avoidance and control. ACM SIGCOMM Comput Commun Rev 18:314–329

  18. 18.

    Jiang N, Kim J, Dally WJ (2009) Indirect adaptive routing on large scale interconnection networks. In: ISCA ’09: 36th international symposium on computer architecture

  19. 19.

    Kerbyson DJ, Barker KJ (2011) Analyzing the performance bottlenecks of the POWER7-IH network. In: CLUSTER. IEEE, pp 244–252

  20. 20.

    Kermani P, Kleinrock L (1976) Virtual cut-through: a new computer communication switching technique. Comput Netw 3(4):267–286

  21. 21.

    Kim J, Dally W, Scott S, Abts D (2008) Technology-driven, highly-scalable dragonfly topology. In: Proceedings of the 35th annual international symposium on computer architecture. IEEE Computer Society, pp 77–88

  22. 22.

    Lam S, Reiser M (1979) Congestion control of store-and-forward networks by input buffer limits—an analysis. IEEE Trans Commun 27(1):127–134. doi:10.1109/TCOM.1979.1094280

  23. 23.

    Pinkston T (2004) Deadlock characterization and resolution in interconnection networks. In: Deadlock resolution in computer-integrated systems, CRC Press, pp 445–492

  24. 24.

    Prisacari B, Rodriguez G, Garcia M, Vallejo E, Beivide R, Minkenberg C (2014) Performance implications of remote-only load balancing under adversarial traffic in dragonflies. In: 8th international workshop on interconnection network architecture: on-chip, multi-chip, INA-OCMC ’14. doi: 10.1145/2556857.2556860

  25. 25.

    Silla F, Duato J (2000) High-performance routing in networks of workstations with irregular topology. IEEE Trans Parallel Distrib Syst 11(7):699–719. doi:10.1109/71.877816

  26. 26.

    Valiant L (1982) A scheme for fast parallel communication. SIAM J Comput 11:350

Download references


This work has been supported by the Spanish Ministry of Education, FPU grant AP2010-4900; the Spanish Science and Technology Commission (CICYT) under contracts TIN2010-21291-C02-02, TIN2012-34557 and TIN2013- 46957-C2-2-P; the European Union FP7 under Agreements ICT-288777 (Mont-Blanc) and ERC-321253 (RoMoL); the European HiPEAC Network of Excellence and the JSA no. 2013-119 as part of the IBM/BSC Technology Center for Supercomputing agreement.

Author information

Correspondence to Enrique Vallejo.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

García, M., Vallejo, E., Beivide, R. et al. On-the-fly adaptive routing for dragonfly interconnection networks. J Supercomput 71, 1116–1142 (2015).

Download citation


  • Interconnection network
  • Dragonfly network
  • OFAR
  • Adaptive routing
  • Deadlock avoidance