The Journal of Supercomputing

, Volume 72, Issue 12, pp 4497–4519 | Cite as

Straightforward solutions to reduce HoL blocking in different Dragonfly fully-connected interconnection patterns

  • Pedro Yébenes
  • Jesus Escudero-Sahuquillo
  • Pedro J. García
  • Francisco J. Quiles


The performance of interconnection networks is a challenging issue for High-Performance Computing (HPC) systems, which becomes even more important when the number of interconnected endnodes grows. In that sense, Dragonfly interconnection patterns are a very popular option to configure the network topology, especially for large systems, as they are able to achieve a high scalability relying on high-radix switches. This kind of hierarchical topologies has two levels of interconnection (i.e., connections within the element of a group and connections among groups) and each one can be interconnected using different patterns. However, regardless of the Dragonfly interconnection pattern, the Head-of-Line (HoL) blocking effect derived from congestion situations may jeopardize the Dragonfly performance. This paper analyzes the dynamics of congestion in different Dragonfly fully-connected interconnection patterns. Also, we describe a queuing scheme called Hierarchical Two-Level Queuing (H2LQ), designed specially to reduce HoL blocking in any fully-connected Dragonfly network that uses minimal-path routing. Finally, we present experiment results which show that this scheme significantly boost Dragonfly performance, regardless the interconnection pattern, especially when congestion arises, while requiring fewer network resources than other techniques oriented to deal with the effects of congestion.


High performance computing Interconnection networks Dragonfly topology HoL blocking Congestion control 


  1. 1.
    The Graph 500 List.
  2. 2.
    VEF traces: An easy way to model MPI traffic in network simulators.
  3. 3.
    Alversons B, Froese E, Kaplan L, Roweth D (2012) Cray XC Series Network. Tech. rep. Cray Inc,Google Scholar
  4. 4.
    Anderson T, Owicki S, Saxe J, Thacker C (1993) High-Speed Switch Scheduling for Local-Area Networks. ACM Transactions on Computer Systems 11(4):319–352CrossRefGoogle Scholar
  5. 5.
    Andujar FJ, Villar JA, Sanchez JL, Alfaro FJ, Escudero-Sahuquillo J (2015) VEF Traces: A Framework for Modelling MPI Traffic in Interconnection Network Simulators. In: Cluster Computing (CLUSTER), 2015 IEEE International Conference on, pp. 841–848. doi: 10.1109/CLUSTER.2015.141
  6. 6.
    Arimilli B, Arimilli R, Chung V, Clark S, Denzel W, Drerup B, Hoefler T, Joyner J, Lewis J, Li J, Ni N, Rajamony R (2010) The PERCS High-Performance Interconnect. In: High Performance Interconnects (HOTI), 2010 IEEE 18th Annual Symposium on, pp. 75–82. doi: 10.1109/HOTI.2010.16
  7. 7.
    Association IT (2007) InfiniBand Architecture Specification.
  8. 8.
    Camarero C, Vallejo E, Beivide R (2014) Topological Characterization of Hamming and Dragonfly Networks and Its Implications on Routing. ACM Trans. Archit. Code Optim. 11(4):39:1–39:25. doi: 10.1145/2677038 CrossRefGoogle Scholar
  9. 9.
    Dally W (1992) Virtual-Channel Flow Control. IEEE Trans. on Parallel and Distributed Systems 3(2):194–205CrossRefGoogle Scholar
  10. 10.
    Dally W, Carvey P, Dennison L (1998) Architecture of the Avici terabit switch/router. In: 6th Hot Interconnects, pp. 41–50Google Scholar
  11. 11.
    Dally WJ, Towles B (2003) Principles and Practices of Interconnection Networks. Morgan Kaufmann Publishers Inc., San Francisco, CA, USAGoogle Scholar
  12. 12.
    Escudero-Sahuquillo J, García PJ, Quiles FJ, Flich J, Duato J (2011) OBQA: Smart and cost-efficient queue scheme for Head-of-Line blocking elimination in fat-trees. J. Parallel Distrib. Comput. 71(11):1460–1472CrossRefGoogle Scholar
  13. 13.
    Escudero-Sahuquillo J, Garcia PJ, Quiles FJ, Reinemo SA, Skeie T, Lysne O, Duato J (2014) A New Proposal to Deal with Congestion in InfiniBand-based Fat-trees. J. Parallel Distrib. Comput. 74(1):1802–1819CrossRefGoogle Scholar
  14. 14.
    Escudero-Sahuquillo J, Gran E, Garcia-Garcia P, Flich J, Skeie T, Lysne O, Quiles F, Duato J (2014) Efficient and Cost-Effective Hybrid Congestion Control for HPC Interconnection Networks. Parallel and Distributed Systems, IEEE Transactions on PP(99):1–1. doi: 10.1109/TPDS.2014.2307851 Google Scholar
  15. 15.
    Garcia P, Quiles F, Flich J, Duato J, Johnson I, Naven F (2006) Efficient, Scalable Congestion Management for Interconnection Networks. Micro, IEEE 26(5):52–66CrossRefGoogle Scholar
  16. 16.
    Gomez C, Gilabert F, Gomez M, Lopez P, Duato J (2007) Deterministic versus Adaptive Routing in Fat-Trees. In: Workshop CAC in conjunction with the IPDPS, p. 235Google Scholar
  17. 17.
    Guay WL, Bogdanski B, Reinemo SA, Lysne O, Skeie T (2011) vFtree - A Fat-Tree Routing Algorithm Using Virtual Lanes to Alleviate Congestion. In: Proc. of IPDPS, pp. 197–208Google Scholar
  18. 18.
    Hastings E, Rincon-Cruz D, Spehlmann M, Meyers S, Xu A, Bunde DP, Leung VJ (2015) Comparing Global Link Arrangements for Dragonfly Networks. In: Cluster Computing (CLUSTER), 2015 IEEE International Conference on, pp. 361–370. doi: 10.1109/CLUSTER.2015.57
  19. 19.
    Jurczyk M, Schwederski T (1996) Phenomenon of Higher Order Head-of-Line Blocking in Multistage Interconnection Networks under Nonuniform Traffic Patterns. IEICE Transactions on Information and Systems E79–D(8):1124–1129Google Scholar
  20. 20.
    Karol MJ, Hluchyj MG, Morgan SP (1987) Input versus output queuing on a space-division packet switch. IEEE Transactions on Communications. COM–35:1347–1356CrossRefGoogle Scholar
  21. 21.
    Katevenis M, Serpanos D, Spyridakis E (1998) Credit-flow-controlled ATM for MP interconnection: The ATLAS I single-chip ATM switch. In: High-Performance Computer Architecture, 1998. Proceedings., 1998 Fourth International Symposium on, pp. 47–56Google Scholar
  22. 22.
    Kim J, Dally WJ, Scott S, Abts D (2008) Technology-Driven, Highly-Scalable Dragonfly Topology. SIGARCH Comput. Archit. News 36(3):77–88CrossRefGoogle Scholar
  23. 23.
    Nachiondo T, Flich J, Duato J (2010) Buffer Management Strategies to Reduce HoL Blocking. Parallel and Distributed Systems, IEEE Transactions on 21(6):739–753. doi: 10.1109/TPDS.2009.63 CrossRefGoogle Scholar
  24. 24.
    Olesinski W, Eberle H, Gura N (2009) Scalable alternatives to virtual output queueing. In: Proc. IEEE ICC, pp. 1–6Google Scholar
  25. 25.
    OpenSim Ltd: OMNeT++ Discrete Event Simulator.
  26. 26.
    Peir JK, Lee YH (1993) Look-ahead routing switches for multistage interconnection networks. Journal of Parallel and Distributed Computing 19(1):1–10. doi: 10.1006/jpdc.1993.1085 CrossRefGoogle Scholar
  27. 27.
    Penaranda R, Gomez C, Gomez M, Lopez P, Duato J (2012) A New Family of Hybrid Topologies for Large-Scale Interconnection Networks. In: Network Computing and Applications (NCA), 2012 11th IEEE International Symposium on, pp. 220–227Google Scholar
  28. 28.
    Pfister G, Gusat M, Denzel W, Craddock D, Ni N, Rooney W, Engbersen T, Luijten R, Krishnamurthy R, Duato J (2005) Solving Hot Spot Contention Using InfiniBand Architecture Congestion Control. In: Proc. of Int. Workshop HPI-DCGoogle Scholar
  29. 29.
    Phillips JC, Braun R, Wang W, Gumbart J, Tajkhorshid E, Villa E, Chipot C, Skeel RD, Kalé L, Schulten K (2005) Scalable molecular dynamics with NAMD. Journal of Computational Chemistry 26(16):1781–1802. doi: 10.1002/jcc.20289 CrossRefGoogle Scholar
  30. 30.
    Pinkston TM, Duato J (2006) Appendix E. In: Elsevier (ed.) Computer Architecture: A Quantitative Approach. Morgan Kaufmann PublishersGoogle Scholar
  31. 31.
    Tamir Y, Frazier G (1992) Dynamically-Allocated Multi-Queue Buffers for VLSI Communication Switches. IEEE Trans. on ComputersGoogle Scholar
  32. 32.
    Yebenes P, Escudero-Sahuquillo J, Garcia P, Quiles F (2013) Towards Modeling Interconnection Networks of Exascale Systems with OMNet++. In: Parallel, Distributed and Network-Based Processing. doi: 10.1109/PDP.2013.36
  33. 33.
    Yebenes Segura P, Escudero-Sahuquillo J, Gomez Requena C, Garcia P, Quiles F, Duato J (2013) BBQ: A Straightforward Queuing Scheme to Reduce HoL-Blocking in High-Performance Hybrid Networks. In: Euro-Par 2013 Parallel Processing, vol. 8097, pp. 699–712Google Scholar
  34. 34.
    Zahavi E, Johnson G, Kerbyson DJ, Lang M (2010) Optimized \(\text{ InfiniBand }^{{\rm TM}}\) fat-tree routing for shift all-to-all communication patterns. Journal of CCPE 22(2):217–231Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Pedro Yébenes
    • 1
  • Jesus Escudero-Sahuquillo
    • 1
  • Pedro J. García
    • 1
  • Francisco J. Quiles
    • 1
  1. 1.Department of Computing SytemsUniversity of Castilla-La Mancha, Instituto de Investigación en Informática de Albacete (I3A)AlbaceteSpain

Personalised recommendations