The Journal of Supercomputing

, Volume 68, Issue 3, pp 1214–1234 | Cite as

Adaptive load balancing in learning-based approaches for many-core embedded systems

  • F. FarahnakianEmail author
  • M. Ebrahimi
  • M. Daneshtalab
  • P. Liljeberg
  • J. Plosila


Adaptive routing algorithms improve network performance by distributing traffic over the whole network. However, they require congestion information to facilitate load balancing. To provide local and global congestion information, we propose a learning method based on dual reinforcement learning approach. This information can be dynamically updated according to the changing traffic condition in the network by propagating data and learning packets. We utilize a congestion detection method which updates the learning rate according to the congestion level. This method calculates the average number of free buffer slots in each switch at specific time intervals and compares it with maximum and minimum values. Based on the comparison result, the learning rate sets to a value between 0 and 1. If a switch gets congested, the learning rate is set to a high value, meaning that the global information is more important than local. In contrast, local is more emphasized than global information in non-congested switches. Results show that the proposed approach achieves a significant performance improvement over the traditional Q-routing, DRQ-routing, DBAR and Dynamic XY algorithms.


Congestion-aware routing algorithm Adaptive routing algorithm Q-learning and Q-routing approaches 


  1. 1.
    Ni LM, McKinley PK (1993) A survey of wormhole routing techniques in direct networks. Computer 26(2):62–76CrossRefGoogle Scholar
  2. 2.
    Ebrahimi M, Daneshtalab M, Liljeberg P, Plosila J, Tenhunen H (2011) Agent-based on-chip network using efficient selection method. In: Proceedings of 19th IFIP/IEEE International Conference on very large scale integration (VLSI-SoC), pp 284–289.Google Scholar
  3. 3.
    Dehyadegari M et al. (2011) An adaptive fuzzy logic-based routing algorithm for networks-on-chip. In: Proceedings of 13th IEEE/NASA-ESA International Conference on adaptive hardware and systems (AHS), pp 208–214.Google Scholar
  4. 4.
    Sutton RS, Barto AG (2000) Reinforcement learning. MIT Press, Cambridge, An introductionGoogle Scholar
  5. 5.
    Watkins CJCH, Dayan P (1992) Q-Learning. In: Proceedings on machine learning, pp 279–292.Google Scholar
  6. 6.
    Boyan JA, Littman ML (1994) Packet routing in dynamically changing networks: a reinforcement learning approach. Adv Neural Inf Process Syst 6:671–678Google Scholar
  7. 7.
    Kumar S, Miikkulainen R (1997) Dual reinforcement Q-routing: an on-line adaptive routing algorithm. In: Proceedings of the artificial neural networks in engineering Conference, pp 231–238.Google Scholar
  8. 8.
    Schonwald T, Zimmermann J, Bringmann O (2007) Fully adaptive fault-tolerant routing algorithm for network-on-chip architectures. In Euromicro Conference on digital system design architectures, methods and tools (DSD), Lübeck, pp 527–534.Google Scholar
  9. 9.
    Chiu G-M (2000) The odd-even turn model for adaptive routing. IEEE Trans Parallel Distrib Syst 11(7):729–738CrossRefGoogle Scholar
  10. 10.
    Ebrahimi M et al. (2012) MAFA: adaptive fault-tolerant routing algorithm for networks-on-chip. In: Proceedings of 15th IEEE Euromicro Conference on Digital System Design (DSD), pp 201–206.Google Scholar
  11. 11.
    Boura YM, Das CR (1994) Efficient fully adaptive wormhole routing in n-dimensional meshes. In: Proceedings of the 14th international conference on distributed computing systems (ICDCS). Pozman, pp 589–596.Google Scholar
  12. 12.
    Feng W, Shin KG (1997) Impact of selection functions on routing algorithm performance in multicomputer networks. In: International Conference on Supercomputing, pp 132–139.Google Scholar
  13. 13.
    Badr HG, Podar S (1989) An optimal shortest-path routing policy for network computers with regular mesh-connected topologies. IEEE Trans Parallel Distrib Syst 38(10):1362–1371MathSciNetGoogle Scholar
  14. 14.
    Li M, Zeng Q, Jone W (2006) DyXY–a proximity congestion-aware deadlock-free dynamic routing method for network on chip. In: Processing of design automation conference (DAC). San Francisco, pp 849–852.Google Scholar
  15. 15.
    Hu J, Marculescu R (2004) DyAD–Smart routing for network-on-chip. In: Processing of design automation conference (DAC). San Diego, pp 260–263.Google Scholar
  16. 16.
    Dally WJ, Aoki H (1993) Deadlock-free adaptive routing in multicomputer networks using virtual channels. IEEE Trans Parallel Distrib Syst 4(4):466–475CrossRefGoogle Scholar
  17. 17.
    Singh A, Dally WJ, Gupta AK, Towles B (2003) GOAL: A load-balanced adaptive routing algorithm for torus networks. In: International Symposium on Computer, Architecture, pp 194–205.Google Scholar
  18. 18.
    Kim J, Park D, Theocharides T, Vijaykrishnan N, Das CR (2005) A low latency router supporting adaptivity for on-chip interconnects. Proceedings of the 42nd annual design automation conference (DAC). ACM, New York, pp 559–564CrossRefGoogle Scholar
  19. 19.
    Ascia G (2008), Implementation and analysis of a new selection strategy for adaptive routing in networks-on-chip. IEEE Trans Comput 57(I.6):809–820.Google Scholar
  20. 20.
    Gratz P, Grot B, Keckler SW (2008) Regional congestion awareness for load balance in networks-on-chip. In: Proceeding of the 14th international symposium on high-performance computer architecture. Salt Lake, City, pp 203–214.Google Scholar
  21. 21.
    Ma S et al. (2011) DBAR: an efficient routing algorithm to support multiple concurrent applications in networks-on-chip. In: Proceeding of 38th annual international symposium on computer architecture (ISCA). San Jose, pp 413–424.Google Scholar
  22. 22.
    Ebrahimi M et al. (2012) CATRA—congestion aware trapezoid-based routing algorithm for on-chip networks. In: Proceeding of design, automation & test in Europe conference & exhibition (DATE). Dresden, pp 320–325.Google Scholar
  23. 23.
    Choi SP, Yeung D-Y (1996) Predictive Q-routing: a memory-based reinforcement learning approach to adaptive traffic control. Adv Neural Inf Process Syst 8(NIPS8):945–951Google Scholar
  24. 24.
    Kumar S, Miikkulainen R (1998) Confidence-based Q-routing: an on-line adaptive network routing algorithm. Smart engineering systems: neural networks, fuzzy logic, data mining, and evolutionary programming 8:147–152Google Scholar
  25. 25.
    Kumar S, Miikkulainen R (1997), Dual reinforcement Q-routing: an on-line adaptive routing algorithm.In: Proceedings of the Artificial Neural Networks in, Engineering Conference, pp 231–238.Google Scholar
  26. 26.
    Feng C, Lu Z, Jantsch A, Li J, Zhang M (2010) A reconfigurable fault-tolerant deflection routing algorithm based on reinforcement learning for network-on-chip. In: Proceedings of NoCArc, pp 11–16.Google Scholar
  27. 27.
    Majer M et al. (2005) Packet routing in dynamically changing networks on chip. In: Proceedings of the 19th International Parallel and Distributed Processing Symposium (IPDPS). Denver, USA.Google Scholar
  28. 28.
    Paliwal KK, George JS, Rameshan N, Laxmi V, Gaur MS, Janyani V, Narasimhan R (2009) Implementation of QoS aware Q-routing algorithm for network-on-chip. In: Contemporary computing. Springer, Berlin, Heidelberg, pp 370–380Google Scholar
  29. 29.
    Puthal MK, Singh V, Gaur MS, Laxmi V (2011) C-routing: an adaptive hierarchical NoC routing methodology. In: IEEE/IFIP 19th international conference on VLSI and system-on-chip (VLSI-SoC). Hong Kong, pp 392–397.Google Scholar
  30. 30.
    Farahnakian F, Ebrahimi M, Daneshtalab M, Liljeberg P, Plosila J (2011) Q-learning based congestion-aware routing algorithm for on-chip network. In: Proceedings of 2nd IEEE international conference on networked embedded systems for enterprise applications (NESEA). Fremantle, pp 1–7.Google Scholar
  31. 31.
    Farahnakian F, Ebrahimi M, Daneshtalab M, Plosila J, Liljeberg P (2012) Adaptive reinforcement learning method for networks-on-chip. In: Proceedings of 12th IEEE international conference on embedded computer systems: architectures, modeling, and simulation (SAMOS XII). Samos, pp 236–243.Google Scholar
  32. 32.
    Ebrahimi M, Daneshtalab M, Farahnakian F, Liljeberg P, Plosila J, Palesi M, Tenhunen H (2012) HARAQ: congestion-aware learning model for highly adaptive routing algorithm in on-chip networks. In: Proceedings of 6th ACM/IEEE International Symposium on Networks-on-Chip (NOCS), pp 19–26.Google Scholar
  33. 33.
    Farahnakian F, Ebrahimi M, Daneshtalab M, Liljeberg P, Plosila J (2014) Bi-LCQ: a low-weight clustering-based Q-learning approach for NoCs. Elsevier J Microprocess Microsyst (MICPRO) 38:64–75CrossRefGoogle Scholar
  34. 34.
    Farahnakian F, Ebrahimi M, Daneshtalab M, Liljeberg P, Plosila J (2012) Optimized Q-learning model for distributing traffic in on-chip networks. In: International Conference on Networked Embedded Systems for Enterprise Applications (NESEA), UK, pp 1–8.Google Scholar
  35. 35.
    Varga A et al. (2001) The OMNeT++ discrete event simulation system. In: Proceedings of the European Simulation Multiconference (ESM’2001), pp 319–324.Google Scholar
  36. 36.
    Woo SC et al. (1995) The splash-2 programs: characterization and methodological considerations. In: Proceedings of Computer Architecture (ISCA), pp 24–36.Google Scholar
  37. 37.
    Martin MK, Sorin DJ, Beckmann BM et al (2005) Multifacet’s general execution driven multiprocessor simulator (GEMS) toolset. SIGARCH Comput Archit News 33(4):92–99Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • F. Farahnakian
    • 1
    Email author
  • M. Ebrahimi
    • 1
  • M. Daneshtalab
    • 1
  • P. Liljeberg
    • 1
  • J. Plosila
    • 1
  1. 1.University of TurkuTurkuFinland

Personalised recommendations