Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture

  • P. J. García
  • J. Flich
  • J. Duato
  • I. Johnson
  • F. J. Quiles
  • F. Naven
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3793)


Designers of large parallel computers and clusters are becoming increasingly concerned with the cost and power consumption of the interconnection network. A simple way to reduce them consists of reducing the number of network components and increasing their utilization. However, doing so without a suitable congestion management mechanism may lead to dramatic throughput degradation when the network enters saturation. Congestion management strategies for lossy networks (computer networks) are well known, but relatively little effort has been devoted to congestion management in lossless networks (parallel computers, clusters, and on-chip networks). Additionally, congestion is much more difficult to solve in this context due to the formation of congestion trees.

In this paper we study the dynamic evolution of congestion trees. We show that, contrary to the common belief, trees do not only grow from the root toward the leaves. There exist cases where trees grow from the leaves to the root, cases where several congestion trees grow independently and later merge, and even cases where some congestion trees completely overlap while being independent. This complex evolution and its implications on switch architecture are analyzed, proposing enhancements to a recently proposed congestion management mechanism and showing the impact on performance of different design decisions.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Almasi, G.S., Gottlieb, A.: Highly parallel computing. Ed. Benjamin-Cummings Publishing Co., Inc. (1994)Google Scholar
  2. 2.
    Anderson, T., Owicki, S., Saxe, J., Thacker, C.: High-Speed Switch Scheduling for Local-Area Networks. ACM Trans. on Computer Systems 11(4), 319–352 (1993)CrossRefGoogle Scholar
  3. 3.
    Baydal, E., Lopez, P., Duato, J.: A Congestion Control Mechanism for Wormhole Networks. In: Proc. 9th. Euromicro Workshop Parallel & Distributed Processing, February 2001, pp. 19–26 (2001)Google Scholar
  4. 4.
    Baydal, E., Lopez, P.: A Robust Mechanism for Congestion Control: INC. In: Proc. 9th International Euro-Par Conference, August 2003, pp. 958–968 (2003)Google Scholar
  5. 5.
    Bianchini, R., LeBlanc, T.J., Kontothanassis, L.I., Crovella, M.E.: Alleviating Memory Contention in Matrix Computations on Large-Scale Shared-Memory Multiprocessors, Technical report 449, Dept. of Computer Science, Rochester University (April 1993)Google Scholar
  6. 6.
    Dally, W.J., Carvey, P., Dennison, L.: The Avici Terabit Switch/Router. In: Proc. Hot Interconnects 6 (August 1998)Google Scholar
  7. 7.
    Dally, W.J.: Virtual-channel flow control. IEEE Trans. on Parallel and Distributed Systems 3(2), 194–205 (1992)CrossRefGoogle Scholar
  8. 8.
    Dally, W.J., Aoki, H.: Deadlock-Free Adaptive Routing in Multicomputer Networks Using Virtual Channels. IEEE Trans. on Parallel and Distributed Systems 4(4), 466–475 (1993)CrossRefGoogle Scholar
  9. 9.
    Dandamudi, S.P.: Reducing Hot-Spot Contention in Shared-Memory Multiprocessor Systems. IEEE Concurrency 7(1), 48–59 (1999)CrossRefGoogle Scholar
  10. 10.
    Duato, J.: A New Theory of Deadlock-Free Adaptive Routing in Wormhole Networks. IEEE Trans. on Parallel and Distributed Systems 4(12), 1320–1331 (1993)CrossRefGoogle Scholar
  11. 11.
    Duato, J., Yalamanchili, S., Ni, L.M.: Interconnection Networks: An Engineering Approach. Morgan Kaufmann Publishers, San Francisco (2003) (Revised printing)Google Scholar
  12. 12.
    Duato, J., Johnson, I., Flich, J., Naven, F., Garcia, P.J., Nachiondo, T.: A New Scalable and Cost-Effective Congestion Management Strategy for Lossless Multistage Interconnection Networks. In: Proc. 11th International Symposium on High-Performance Computer Architecture (HPCA 2005), February 2005, pp. 108–119 (2005)Google Scholar
  13. 13.
    Franco, D., Garces, I., Luque, E.: A New Method to Make Communication Latency Uniform: Distributed Routing Balancing. In: Proc. ACM International Conference on Supercomputing (ICS 1999), May 1999, pp. 210–219 (1999)Google Scholar
  14. 14.
    Gaughan, P.T., Yalamanchili, S.: Adaptive Routing Protocols for Hypercube Interconnection Networks. IEEE Computer 26(5), 12–23 (1993)Google Scholar
  15. 15.
    Ho, W.S., Eager, D.L.: A Novel Strategy for Controlling Hot Spot Contention. In: Proc. Int. Conf. Parallel Processing, vol. I, pp. 14–18 (1989)Google Scholar
  16. 16.
    InfiniBand Trade Association: InfiniBand Architecture. Specification Volume 1. Release 1.0, Available at http://www.infinibandta.com/
  17. 17.
    Karol, M., Hluchyj, M., Morgen, S.: Input versus Output Queueing on a Space Division Switch. IEEE Transactions on Communications 35(12), 1347–1356 (1987)CrossRefGoogle Scholar
  18. 18.
    Katevenis, M., Serpanos, D., Spyridakis, E.: Credit-Flow-Controlled ATM for MP Interconnection: the ATLAS I Single-Chip ATM Switch. In: Proc. 4th Int. Symp. on High-Performance Computer Architecture, February 1998, pp. 47–56 (1998)Google Scholar
  19. 19.
    Kim, J.H., Liu, Z., Chien, A.A.: Compressionless Routing: A Framework for Adaptive and Fault-Tolerant Routing. IEEE Trans. on Parallel and Distributed Systems 8(3) (1997)Google Scholar
  20. 20.
    Konstantinidou, S., Snyder, L.: Chaos Router: Architecture and Performance. In: Proc. 18th International Symposium on Computer Architecture, June 1991, pp. 79–88 (1991)Google Scholar
  21. 21.
    Krishnan, V., Mayhew, D.: A Localized Congestion Control Mechanism for PCI Express Advanced Switching Fabrics. In: Proc. 12th IEEE Symp. on Hot Interconnects (August 2004)Google Scholar
  22. 22.
    Liu, J., Shin, K.G., Chang, C.C.: Prevention of Congestion in Packet-Switched Multistage Interconnection Networks. IEEE Transactions on Parallel Distributed Systems 6(5), 535–541 (1995)CrossRefGoogle Scholar
  23. 23.
    Lopez, P., Duato, J.: Deadlock-Free Adaptive Routing Algorithms for the 3D-Torus: Limitations and Solutions. In: Proc. Parallel Architectures and Languages Europe 93 (June 1993)Google Scholar
  24. 24.
  25. 25.
    Advanced Switching for the PCI Express Architecture. White paper, Available at http://www.intel.com/technology/pciexpress/devnet/AdvancedSwitching.pdf
  26. 26.
    Advanced Switching Core Architecture Specification, Available at http://www.asi-sig.org/ specifications for ASI SIG
  27. 27.
    Pfister, G., Norton, A.: Hot Spot Contention and Combining in Multistage Interconnect Networks. IEEE Trans. on Computers C-34, 943–948 (1985)Google Scholar
  28. 28.
    Quadrics QsNet, Available at http://doc.quadrics.com
  29. 29.
    Scott, S.L., Sohi, G.S.: The Use of Feedback in Multiprocessors and Its Application to Tree Saturation Control. IEEE Transactions on Parallel Distributed Systems 1(4), 385–398 (1990)CrossRefGoogle Scholar
  30. 30.
    Shang, L., Peh, L.S., Jha, N.K.: Dynamic Voltage Scaling with Links for Power Optimization of Interconnection Networks. In: Proc. Int. Symp. on High-Performance Computer Architecture, February 2003, pp. 91–102 (2003)Google Scholar
  31. 31.
    Singh, A., Dally, W.J., Towles, B., Gupta, A.K.: Globally Adaptive Load-Balanced Routing on Tori. Computer Architecture Letters 3(1), 6–9 (2004)CrossRefGoogle Scholar
  32. 32.
    Smai, A., Thorelli, L.: Global Reactive Congestion Control in Multicomputer Networks. In: Proc. 5th Int. Conf. on High Performance Computing (1998)Google Scholar
  33. 33.
  34. 34.
    Stine, J.M., Carter, N.P.: Comparing Adaptive Routing and Dynamic Voltage Scaling for Link Power Reduction. Computer Architecture Letters 3(1), 14–17 (2004)Google Scholar
  35. 35.
    Tamir, Y., Frazier, G.L.: Dynamically-Allocated Multi-Queue Buffers for VLSI Communication Switches. IEEE Trans. on Computers 41(6) (June 1992)Google Scholar
  36. 36.
    Thottethodi, M., Lebeck, A.R., Mukherjee, S.S.: Self-Tuned Congestion Control for Multiprocessor Networks. In: Proc. Int. Symp. High-Performance Computer Architecture (February 2001)Google Scholar
  37. 37.
    Thottethodi, M., Lebeck, A.R., Mukherjee, S.S.: BLAM: A High-Performance Routing Algorithm for Virtual Cut-Through Networks. In: Proc. Int. Parallel and Distributed Processing Symp (IPDPS) (April 2003)Google Scholar
  38. 38.
    Vogels, W., et al.: Tree-Saturation Control in the AC3 Velocity Cluster Interconnect. In: Proc. 8th Conference on Hot Interconnects (August 2000)Google Scholar
  39. 39.
    Wang, M., Siegel, H.J., Nichols, M.A., Abraham, S.: Using a Multipath Network for Reducing the Effects of Hot Spots. IEEE Transactions on Parallel and Distributed Systems 6(3), 252–268 (1995)CrossRefGoogle Scholar
  40. 40.
    Yang, C.Q., Reddy, A.V.S.: A Taxonomy for Congestion Control Algorithms in Packet Switching Networks. IEEE Network, 34–45 (July/August 1995)Google Scholar
  41. 41.
    Yew, P., Tzeng, N., Lawrie, D.H.: Distributing Hot-Spot Addressing in Large-Scale Multiprocessors. IEEE Transactions on Computers 36(4), 388–395 (1987)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • P. J. García
    • 1
  • J. Flich
    • 2
  • J. Duato
    • 2
  • I. Johnson
    • 3
  • F. J. Quiles
    • 1
  • F. Naven
    • 3
  1. 1.Dept. de InformáticaUniv. Castilla-La ManchaAlbaceteSpain
  2. 2.Dept. of Computer ScienceUniv. Politécnica de ValenciaValenciaSpain
  3. 3.XyratexHavenUnited Kingdom

Personalised recommendations